diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md index 8ca10109b..49089191f 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.md +++ b/.github/ISSUE_TEMPLATE/bug_report.md @@ -12,7 +12,7 @@ The GitHub issue tracker's purpose is to deal with bug reports and feature reque Keep in mind that by filing an issue, you are expected to comply with Scrapy's Code of Conduct, including treating everyone with respect: https://github.com/scrapy/scrapy/blob/master/CODE_OF_CONDUCT.md -The following is a suggested template to structure your issue, you can find more guidelines at https://doc.scrapy.org/en/latest/contributing.html#reporting-bugs +The following is a suggested template to structure your issue, you can find more guidelines at https://docs.scrapy.org/en/latest/contributing.html#reporting-bugs --> diff --git a/.github/ISSUE_TEMPLATE/feature_request.md b/.github/ISSUE_TEMPLATE/feature_request.md index e05273fe2..7d9995de6 100644 --- a/.github/ISSUE_TEMPLATE/feature_request.md +++ b/.github/ISSUE_TEMPLATE/feature_request.md @@ -12,7 +12,7 @@ The GitHub issue tracker's purpose is to deal with bug reports and feature reque Keep in mind that by filing an issue, you are expected to comply with Scrapy's Code of Conduct, including treating everyone with respect: https://github.com/scrapy/scrapy/blob/master/CODE_OF_CONDUCT.md -The following is a suggested template to structure your pull request, you can find more guidelines at https://doc.scrapy.org/en/latest/contributing.html#writing-patches and https://doc.scrapy.org/en/latest/contributing.html#submitting-patches +The following is a suggested template to structure your pull request, you can find more guidelines at https://docs.scrapy.org/en/latest/contributing.html#writing-patches and https://docs.scrapy.org/en/latest/contributing.html#submitting-patches --> diff --git a/docs/contributing.rst b/docs/contributing.rst index 6d2e08fe8..563d52283 100644 --- a/docs/contributing.rst +++ b/docs/contributing.rst @@ -6,7 +6,7 @@ Contributing to Scrapy .. important:: - Double check that you are reading the most recent version of this document + Double-check that you are reading the most recent version of this document at https://docs.scrapy.org/en/master/contributing.html By participating in this project you agree to abide by the terms of our @@ -216,14 +216,13 @@ has been validated and proven useful. Alternatively, you can start a conversation in the `Scrapy subreddit`_ to discuss your idea first. Sometimes there is an existing pull request for the problem you'd like to -solve, which is stalled for some reason. Often the pull request is in a -right direction, but changes are requested by Scrapy maintainers, and the -original pull request author hasn't had time to address them. -In this case consider picking up this pull request: open -a new pull request with all commits from the original pull request, as well as -additional changes to address the raised issues. Doing so helps a lot; it is -not considered rude as long as the original author is acknowledged by keeping -his/her commits. +solve, which is stalled for some reason. Often the pull request is in the right +direction, but changes are requested by Scrapy maintainers, and the original +pull request author hasn't had time to address them. In this case consider +picking up this pull request: open a new pull request with all commits from the +original pull request, as well as additional changes to address the raised +issues. Doing so helps a lot; it is not considered rude as long as the original +author is acknowledged by keeping their commits. You can pull an existing pull request to a local branch by running ``git fetch upstream pull/$PR_NUMBER/head:$BRANCH_NAME_TO_CREATE`` @@ -233,9 +232,9 @@ with a name of the branch you want to create locally). See also: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally#modifying-an-inactive-pull-request-locally. When writing GitHub pull requests, try to keep titles short but descriptive. -E.g. For bug #411: "Scrapy hangs if an exception raises in start_requests" -prefer "Fix hanging when exception occurs in start_requests (#411)" -instead of "Fix for #411". Complete titles make it easy to skim through +For example, for bug #411: "Scrapy hangs if an exception raises in +start_requests" prefer "Fix hanging when exception occurs in start_requests +(#411)" instead of "Fix for #411". Complete titles make it easy to skim through the issue tracker. Finally, try to keep aesthetic changes (:pep:`8` compliance, unused imports @@ -271,7 +270,7 @@ commit. .. _pre-commit: https://pre-commit.com/ -After your create a local clone of your fork of the Scrapy repository: +After you create a local clone of your fork of the Scrapy repository: #. `Install pre-commit `_. diff --git a/docs/faq.rst b/docs/faq.rst index 73a72a2df..f859597e5 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -118,7 +118,7 @@ My Scrapy crawler has memory leaks. What can I do? See :ref:`topics-leaks`. -Also, Python has a builtin memory leak issue which is described in +Also, Python has a built-in memory leak issue which is described in :ref:`topics-leaks-without-leaks`. How can I make Scrapy consume less memory? @@ -136,8 +136,8 @@ middleware with a :ref:`custom downloader middleware ` that requires less memory. For example: - If your domain names are similar enough, use your own regular expression - instead of joining the strings in :attr:`~scrapy.Spider.allowed_domains` into - a complex regular expression. + instead of joining the strings in :attr:`~scrapy.Spider.allowed_domains` + into a complex regular expression. - If you can meet the installation requirements, use pyre2_ instead of Python’s re_ to compile your URL-filtering regular expression. See @@ -276,8 +276,8 @@ Parsing big feeds with XPath selectors can be problematic since they need to build the DOM of the entire feed in memory, and this can be quite slow and consume a lot of memory. -In order to avoid parsing all the entire feed at once in memory, you can use -the :func:`~scrapy.utils.iterators.xmliter_lxml` and +In order to avoid parsing the entire feed at once in memory, you can use the +:func:`~scrapy.utils.iterators.xmliter_lxml` and :func:`~scrapy.utils.iterators.csviter` functions. In fact, this is what :class:`~scrapy.spiders.XMLFeedSpider` and :class:`~scrapy.spiders.CSVFeedSpider` use. @@ -314,8 +314,8 @@ Should I use spider arguments or settings to configure my spider? ----------------------------------------------------------------- Both :ref:`spider arguments ` and :ref:`settings ` -can be used to configure your spider. There is no strict rule that mandates to -use one or the other, but settings are more suited for parameters that, once +can be used to configure your spider. There is no strict rule that mandates +using one or the other, but settings are more suited for parameters that, once set, don't change much, while spider arguments are meant to change more often, even on each spider run and sometimes are required for the spider to run at all (for example, to set the start url of a spider). diff --git a/docs/index.rst b/docs/index.rst index de06e3488..989613c60 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -21,7 +21,7 @@ Having trouble? We'd like to help! * Try the :doc:`FAQ ` -- it's got answers to some common questions. * Looking for specific information? Try the :ref:`genindex` or :ref:`modindex`. -* Ask or search questions in `StackOverflow using the scrapy tag`_. +* Ask or search questions on `Stack Overflow using the scrapy tag`_. * Ask or search questions in the `Scrapy subreddit`_. * Search for questions on the archives of the `scrapy-users mailing list`_. * Ask a question in the `#scrapy IRC channel`_. @@ -30,7 +30,7 @@ Having trouble? We'd like to help! .. _scrapy-users mailing list: https://groups.google.com/forum/#!forum/scrapy-users .. _Scrapy subreddit: https://www.reddit.com/r/scrapy/ -.. _StackOverflow using the scrapy tag: https://stackoverflow.com/tags/scrapy +.. _Stack Overflow using the scrapy tag: https://stackoverflow.com/tags/scrapy .. _#scrapy IRC channel: irc://irc.freenode.net/scrapy .. _issue tracker: https://github.com/scrapy/scrapy/issues .. _Scrapy Discord: https://discord.com/invite/mv3yErfpvq @@ -135,7 +135,7 @@ Built-in services topics/telnetconsole :doc:`topics/logging` - Learn how to use Python's built-in logging on Scrapy. + Learn how to use Python's built-in logging in Scrapy. :doc:`topics/stats` Collect statistics about your scraping crawler. @@ -200,7 +200,7 @@ Solving specific problems Download files and/or images associated with your scraped items. :doc:`topics/deploy` - Deploying your Scrapy spiders and run them in a remote server. + Deploy your Scrapy spiders and run them on a remote server. :doc:`topics/autothrottle` Adjust crawl rate dynamically based on load. @@ -252,7 +252,7 @@ Extending Scrapy Customize the input and output of your spiders. :doc:`topics/extensions` - Extend Scrapy with your custom functionality + Extend Scrapy with your custom functionality. :doc:`topics/signals` See all available signals and how to work with them. @@ -261,7 +261,7 @@ Extending Scrapy Understand the scheduler component. :doc:`topics/exporters` - Quickly export your scraped items to a file (XML, CSV, etc). + Quickly export your scraped items to a file (XML, CSV, etc.). :doc:`topics/download-handlers` Customize how requests are downloaded or add support for new URL schemes. @@ -271,7 +271,7 @@ Extending Scrapy components. :doc:`topics/api` - Use it on extensions and middlewares to extend Scrapy functionality. + Use it in extensions and middlewares to extend Scrapy functionality. All the rest diff --git a/docs/intro/examples.rst b/docs/intro/examples.rst index edff894c6..f72373577 100644 --- a/docs/intro/examples.rst +++ b/docs/intro/examples.rst @@ -5,16 +5,16 @@ Examples ======== The best way to learn is with examples, and Scrapy is no exception. For this -reason, there is an example Scrapy project named quotesbot_, that you can use to +reason, there is an example Scrapy project named quotesbot_ that you can use to play and learn more about Scrapy. It contains two spiders for -https://quotes.toscrape.com, one using CSS selectors and another one using XPath -expressions. +https://quotes.toscrape.com, one using CSS selectors and another one using +XPath expressions. The quotesbot_ project is available at: https://github.com/scrapy/quotesbot. You can find more information about it in the project's README. -If you're familiar with git, you can checkout the code. Otherwise you can -download the project as a zip file by clicking -`here `_. +If you're familiar with git, you can check out the code. Otherwise, you can +download the project as a zip file by clicking `here +`_. .. _quotesbot: https://github.com/scrapy/quotesbot diff --git a/docs/intro/install.rst b/docs/intro/install.rst index 866c5abe2..3cd13ca5f 100644 --- a/docs/intro/install.rst +++ b/docs/intro/install.rst @@ -25,15 +25,15 @@ To install Scrapy using ``conda``, run:: conda install -c conda-forge scrapy -Alternatively, if you’re already familiar with installation of Python packages, -you can install Scrapy and its dependencies from PyPI with:: +Alternatively, if you’re already familiar with installing Python packages, you +can install Scrapy and its dependencies from PyPI with:: pip install Scrapy We strongly recommend that you install Scrapy in :ref:`a dedicated virtualenv `, to avoid conflicting with your system packages. -Note that sometimes this may require solving compilation issues for some Scrapy +Note that this may sometimes require solving compilation issues for some Scrapy dependencies depending on your operating system, so be sure to check the :ref:`intro-install-platform-notes`. @@ -47,7 +47,7 @@ Things that are good to know Scrapy is written in pure Python and depends on a few key Python packages (among others): * `lxml`_, an efficient XML and HTML parser -* `parsel`_, an HTML/XML data extraction library written on top of lxml, +* `parsel`_, an HTML/XML data extraction library written on top of lxml * `w3lib`_, a multi-purpose helper for dealing with URLs and web page encodings * `twisted`_, an asynchronous networking framework * `cryptography`_ and `pyOpenSSL`_, to deal with various network-level security needs @@ -73,14 +73,14 @@ Using a virtual environment (recommended) TL;DR: We recommend installing Scrapy inside a virtual environment on all platforms. -Python packages can be installed either globally (a.k.a system wide), -or in user-space. We do not recommend installing Scrapy system wide. +Python packages can be installed either globally (a.k.a. system-wide), or in +user-space. We do not recommend installing Scrapy system-wide. -Instead, we recommend that you install Scrapy within a so-called -"virtual environment" (:mod:`venv`). -Virtual environments allow you to not conflict with already-installed Python -system packages (which could break some of your system tools and scripts), -and still install packages normally with ``pip`` (without ``sudo`` and the likes). +Instead, we recommend that you install Scrapy within a so-called "virtual +environment" (:mod:`venv`). Virtual environments allow you to avoid conflicts +with already-installed Python system packages (which could break some of your +system tools and scripts), and still install packages normally with ``pip`` +(without ``sudo`` or similar tools). See :ref:`tut-venv` on how to create your virtual environment. @@ -140,7 +140,7 @@ The following extras are available: .. _intro-install-platform-notes: -Platform specific installation notes +Platform-specific installation notes ==================================== .. _intro-install-windows: @@ -168,11 +168,12 @@ To install Scrapy on Windows using ``pip``: #. Under the Workloads section, select **C++ build tools**. -#. Check the installation details and make sure following packages are selected as optional components: +#. Check the installation details and make sure the following packages are + selected as optional components: - * **MSVC** (e.g MSVC v142 - VS 2019 C++ x64/x86 build tools (v14.23) ) + * **MSVC** (e.g. MSVC v142 - VS 2019 C++ x64/x86 build tools (v14.23) ) - * **Windows SDK** (e.g Windows 10 SDK (10.0.18362.0)) + * **Windows SDK** (e.g. Windows 10 SDK (10.0.18362.0)) #. Install the Visual Studio Build Tools. @@ -188,8 +189,8 @@ twisted and pyOpenSSL, and is compatible with recent Ubuntu distributions. But it should support older versions of Ubuntu too, like Ubuntu 14.04, albeit with potential issues with TLS connections. -**Don't** use the ``python-scrapy`` package provided by Ubuntu, they are -typically too old and slow to catch up with the latest Scrapy release. +**Don't** use the ``python-scrapy`` package provided by Ubuntu; it is typically +too old and slow to catch up with the latest Scrapy release. To install Scrapy on Ubuntu (or Ubuntu-based) systems, you need to install @@ -235,8 +236,8 @@ solutions: * Install `homebrew`_ following the instructions in https://brew.sh/ * Update your ``PATH`` variable to state that homebrew packages should be - used before system packages (Change ``.bashrc`` to ``.zshrc`` accordingly - if you're using `zsh`_ as default shell):: + used before system packages (change ``.bashrc`` to ``.zshrc`` accordingly + if you're using `zsh`_ as the default shell):: echo "export PATH=/usr/local/bin:/usr/local/sbin:$PATH" >> ~/.bashrc @@ -244,7 +245,7 @@ solutions: source ~/.bashrc - * Install python:: + * Install Python:: brew install python @@ -254,7 +255,7 @@ solutions: This method is a workaround for the above macOS issue, but it's an overall good practice for managing dependencies and can complement the first method. -After any of these workarounds you should be able to install Scrapy:: +After any of these workarounds, you should be able to install Scrapy:: pip install Scrapy @@ -266,14 +267,14 @@ We recommend using the latest PyPy version. For PyPy3, only Linux installation was tested. Most Scrapy dependencies now have binary wheels for CPython, but not for PyPy. -This means that these dependencies will be built during installation. -On macOS, you are likely to face an issue with building the Cryptography -dependency. The solution to this problem is described -`here `_, +This means that these dependencies will be built during installation. On macOS, +you are likely to face an issue with building the Cryptography dependency. The +solution to this problem is described `here +`_, that is to ``brew install openssl`` and then export the flags that this command -recommends (only needed when installing Scrapy). Installing on Linux has no special -issues besides installing build dependencies. -Installing Scrapy with PyPy on Windows is not tested. +recommends (only needed when installing Scrapy). Installing Scrapy on Linux has +no special issues beyond installing the build dependencies. Installing Scrapy +with PyPy on Windows has not been tested. You can check that Scrapy is installed correctly by running ``scrapy bench``. If this command gives errors such as diff --git a/docs/intro/overview.rst b/docs/intro/overview.rst index 5937c5860..4ef47ce90 100644 --- a/docs/intro/overview.rst +++ b/docs/intro/overview.rst @@ -4,13 +4,13 @@ Scrapy at a glance ================== -Scrapy (/ˈskreɪpaɪ/) is an application framework for crawling web sites and extracting -structured data which can be used for a wide range of useful applications, like -data mining, information processing or historical archival. +Scrapy (/ˈskreɪpaɪ/) is an application framework for crawling websites and +extracting structured data which can be used for a wide range of useful +applications, like data mining, information processing or historical archival. Even though Scrapy was originally designed for `web scraping`_, it can also be used to extract data using APIs (such as `Amazon Associates Web Services`_) or -as a general purpose web crawler. +as a general-purpose web crawler. Walk-through of an example spider @@ -19,7 +19,7 @@ Walk-through of an example spider In order to show you what Scrapy brings to the table, we'll walk you through an example of a Scrapy Spider using the simplest way to run a spider. -Here's the code for a spider that scrapes famous quotes from website +Here's the code for a spider that scrapes famous quotes from the website https://quotes.toscrape.com, following the pagination: .. code-block:: python @@ -49,8 +49,9 @@ and run the spider using the :command:`runspider` command:: scrapy runspider quotes_spider.py -o quotes.jsonl -When this finishes you will have in the ``quotes.jsonl`` file a list of the -quotes in JSON Lines format, containing the text and author, which will look like this:: +When this finishes, you will have a list of the quotes in JSON Lines format in +the ``quotes.jsonl`` file, containing the text and author, which will look like +this:: {"author": "Jane Austen", "text": "\u201cThe person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.\u201d"} {"author": "Steve Martin", "text": "\u201cA day without sunshine is like, you know, night.\u201d"} @@ -73,19 +74,19 @@ look for a link to the next page and schedule another request using the same ``parse`` method as callback. Here you will notice one of the main advantages of Scrapy: requests are -:ref:`scheduled and processed asynchronously `. This -means that Scrapy doesn't need to wait for a request to be finished and -processed, it can send another request or do other things in the meantime. This -also means that other requests can keep going even if a request fails or an -error happens while handling it. +:ref:`scheduled and processed asynchronously `. This means +that Scrapy doesn't need to wait for a request to be finished and processed; it +can send another request or do other things in the meantime. This also means +that other requests can keep going even if a request fails or an error happens +while handling it. While this enables you to do very fast crawls (sending multiple concurrent requests at the same time, in a fault-tolerant way) Scrapy also gives you control over the politeness of the crawl through :ref:`a few settings `. You can do things like setting a download delay between -each request, limiting the amount of concurrent requests per domain, and -even :ref:`using an auto-throttling extension ` that tries -to figure these settings out automatically. +each request, limiting the number of concurrent requests per domain, and even +:ref:`using an auto-throttling extension ` that tries to +figure these settings out automatically. .. note:: @@ -117,7 +118,7 @@ scraping easy and efficient, such as: multiple formats (JSON, CSV, XML) and storing them in multiple backends (FTP, S3, local filesystem) -* Robust encoding support and auto-detection, for dealing with foreign, +* Robust encoding support and auto-detection for dealing with foreign, non-standard and broken encoding declarations. * :ref:`Strong extensibility support `, allowing you to plug @@ -138,10 +139,10 @@ scraping easy and efficient, such as: console running inside your Scrapy process, to introspect and debug your crawler -* Plus other goodies like reusable spiders to crawl sites from `Sitemaps`_ and - XML/CSV feeds, a media pipeline for :ref:`automatically downloading images - ` (or any other media) associated with the scraped - items, a caching DNS resolver, and much more! +* Plus other goodies such as reusable spiders to crawl sites from `Sitemaps`_ + and XML/CSV feeds, a media pipeline for :ref:`automatically downloading + images ` (or any other media) associated with the + scraped items, a caching DNS resolver, and much more! What's next? ============ diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rst index efade47e6..5a234d149 100644 --- a/docs/intro/tutorial.rst +++ b/docs/intro/tutorial.rst @@ -15,7 +15,7 @@ This tutorial will walk you through these tasks: 1. Creating a new Scrapy project 2. Writing a :ref:`spider ` to crawl a site and extract data 3. Exporting the scraped data using the command line -4. Changing spider to recursively follow links +4. Changing the spider to recursively follow links 5. Using spider arguments Scrapy is written in Python_. The more you learn about Python, the more you @@ -81,10 +81,11 @@ ask you to adjust it, rather than block it. Our first Spider ================ -Spiders are classes that you define and that Scrapy uses to scrape information from a website -(or a group of websites). They must subclass :class:`~scrapy.Spider` and define the initial -requests to be made, and optionally, how to follow links in pages and parse the downloaded -page content to extract data. +Spiders are classes that you define and that Scrapy uses to scrape information +from a website (or a group of websites). They must subclass +:class:`~scrapy.Spider` and define the initial requests to be made and, +optionally, how to follow links in pages and parse the downloaded page content +to extract data. This is the code for our first Spider. Save it in a file named ``quotes_spider.py`` under the ``tutorial/spiders`` directory in your project: @@ -142,9 +143,9 @@ To put our spider to work, go to the project's top level directory and run:: scrapy crawl quotes -This command runs the spider named ``quotes`` that we've just added, that -will send some requests for the ``quotes.toscrape.com`` domain. You will get an output -similar to this:: +This command runs the spider named ``quotes`` that we've just added, which will +send some requests for the ``quotes.toscrape.com`` domain. You will get an +output similar to this:: ... (omitted for brevity) 2016-12-16 21:24:05 [scrapy.core.engine] INFO: Spider opened @@ -405,8 +406,8 @@ like this: -Let's open up scrapy shell and play a bit to find out how to extract the data -we want:: +Let's open up the Scrapy shell and play a bit to find out how to extract the +data we want:: scrapy shell 'https://quotes.toscrape.com' @@ -673,14 +674,14 @@ As a shortcut for creating Request objects you can use if next_page is not None: yield response.follow(next_page, callback=self.parse) -Unlike scrapy.Request, ``response.follow`` supports relative URLs directly - no -need to call urljoin. Note that ``response.follow`` just returns a Request -instance; you still have to yield this Request. +Unlike :class:`scrapy.Request`, ``response.follow`` supports relative URLs +directly - no need to call urljoin. Note that ``response.follow`` just returns +a Request instance; you still have to yield this Request. .. skip: start -You can also pass a selector to ``response.follow`` instead of a string; -this selector should extract necessary attributes: +You can also pass a selector to ``response.follow`` instead of a string; this +selector should extract the necessary attributes: .. code-block:: python @@ -746,7 +747,7 @@ this time for scraping author information: } This spider will start from the main page, it will follow all the links to the -authors pages calling the ``parse_author`` callback for each of them, and also +author pages calling the ``parse_author`` callback for each of them, and also the pagination links with the ``parse`` callback as we saw before. Here we're passing callbacks to @@ -754,8 +755,8 @@ Here we're passing callbacks to arguments to make the code shorter; it also works for :class:`~scrapy.Request`. -The ``parse_author`` callback defines a helper function to extract and cleanup the -data from a CSS query and yields the Python dict with the author data. +The ``parse_author`` callback defines a helper function to extract and clean up +the data from a CSS query and yields the Python dict with the author data. Another interesting thing this spider demonstrates is that, even if there are many quotes from the same author, we don't need to worry about visiting the diff --git a/docs/news.rst b/docs/news.rst index 38624948d..3530cd6bf 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -1921,7 +1921,7 @@ Scrapy 2.13.2 (2025-06-09) when it's closed before being fully initialized. (:gh:`6857`, :gh:`6867`) -- Improved the README, updated the Scrapy logo in it. +- Improved the README and updated the Scrapy logo in it. (:gh:`6831`, :gh:`6833`, :gh:`6839`) - Restricted the Twisted version used in tests to below 25.5.0, as some tests @@ -1940,14 +1940,14 @@ Scrapy 2.13.2 (2025-06-09) Scrapy 2.13.1 (2025-05-28) -------------------------- -- Give callback requests precedence over start requests when priority values +- Gave callback requests precedence over start requests when priority values are the same. - This makes changes from 2.13.0 to start request handling more intuitive and - backward compatible. For scenarios where all requests have the same + This makes the changes from 2.13.0 to start request handling more intuitive + and backward compatible. For scenarios where all requests have the same priorities, in 2.13.0 all start requests were sent before the first - callback request. In 2.13.1, same as in 2.12 and lower, start requests are - only sent when there are not enough pending callback requests to reach + callback request. In 2.13.1, the same as in 2.12 and lower, start requests + are only sent when there are not enough pending callback requests to reach concurrency limits. (:gh:`6828`) diff --git a/docs/topics/addons.rst b/docs/topics/addons.rst index 75f15b3ae..2a0d8522e 100644 --- a/docs/topics/addons.rst +++ b/docs/topics/addons.rst @@ -4,7 +4,7 @@ Add-ons ======= -Scrapy's add-on system is a framework which unifies managing and configuring +Scrapy's add-on system is a framework that unifies managing and configuring components that extend Scrapy's core functionality, such as middlewares, extensions, or pipelines. It provides users with a plug-and-play experience in Scrapy extension management, and grants extensive configuration control to @@ -77,7 +77,7 @@ usually best to leave its priority unchanged. For example, when editing a If the ``update_settings`` method raises :exc:`scrapy.exceptions.NotConfigured`, the add-on will be skipped. This makes -it easy to enable an add-on only when some conditions are met. +it easy to enable an add-on only when certain conditions are met. Fallbacks --------- diff --git a/docs/topics/api.rst b/docs/topics/api.rst index 598edfeb5..b76c7cf5a 100644 --- a/docs/topics/api.rst +++ b/docs/topics/api.rst @@ -22,9 +22,9 @@ functionality into Scrapy. :synopsis: The Scrapy crawler The Extension Manager is responsible for loading and keeping track of installed -extensions and it's configured through the :setting:`EXTENSIONS` setting which -contains a dictionary of all available extensions and their order similar to -how you :ref:`configure the downloader middlewares +extensions, and it's configured through the :setting:`EXTENSIONS` setting, +which contains a dictionary of all available extensions and their order similar +to how you :ref:`configure the downloader middlewares `. .. autoclass:: Crawler @@ -96,7 +96,7 @@ how you :ref:`configure the downloader middlewares The execution engine, which coordinates the core crawling logic between the scheduler, downloader and spiders. - Some extension may want to access the Scrapy engine, to inspect or + Some extensions may want to access the Scrapy engine to inspect or modify the downloader and scheduler behaviour, although this is an advanced use and this API is not yet stable. diff --git a/docs/topics/architecture.rst b/docs/topics/architecture.rst index c60c43f3c..3f0adedfd 100644 --- a/docs/topics/architecture.rst +++ b/docs/topics/architecture.rst @@ -62,9 +62,9 @@ this: :ref:`Spider Middleware ` (see :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_output`). -8. The :ref:`Engine ` sends processed items to - :ref:`Item Pipelines `, then sends processed Requests to - the :ref:`Scheduler ` and asks for possible next Requests +8. The :ref:`Engine ` sends processed items to :ref:`Item + Pipelines `, then sends processed Requests to the + :ref:`Scheduler ` and asks for possible next Requests to crawl. 9. The process repeats (from step 3) until there are no more requests from the diff --git a/docs/topics/benchmarking.rst b/docs/topics/benchmarking.rst index e8ddec00c..9dcb45e89 100644 --- a/docs/topics/benchmarking.rst +++ b/docs/topics/benchmarking.rst @@ -78,8 +78,8 @@ You should see an output like this:: That tells you that Scrapy is able to crawl about 3000 pages per minute in the hardware where you run it. Note that this is a very simple spider intended to follow links, any custom spider you write will probably do more stuff which -results in slower crawl rates. How slower depends on how much your spider does -and how well it's written. +results in slower crawl rates. How much slower depends on how much your spider +does and how well it's written. Use scrapy-bench_ for more complex benchmarking. diff --git a/docs/topics/commands.rst b/docs/topics/commands.rst index 50da4593a..966e8b8d4 100644 --- a/docs/topics/commands.rst +++ b/docs/topics/commands.rst @@ -10,7 +10,7 @@ Scrapy is controlled through the ``scrapy`` command-line tool, to be referred to here as the "Scrapy tool" to differentiate it from the sub-commands, which we just call "commands" or "Scrapy commands". -The Scrapy tool provides several commands, for multiple purposes, and each one +The Scrapy tool provides several commands for multiple purposes, and each one accepts a different set of arguments and options. (The ``scrapy deploy`` command has been removed in 1.0 in favor of the @@ -65,7 +65,7 @@ structure by default, similar to this:: ... The directory where the ``scrapy.cfg`` file resides is known as the *project -root directory*. That file contains the name of the python module that defines +root directory*. That file contains the name of the Python module that defines the project settings. Here is an example: .. code-block:: ini @@ -105,7 +105,7 @@ for ``scrapy`` to use:: Using the ``scrapy`` tool ========================= -You can start by running the Scrapy tool with no arguments and it will print +You can start by running the Scrapy tool with no arguments, and it will print some usage help and the available commands:: Scrapy X.Y - no active project @@ -119,8 +119,8 @@ some usage help and the available commands:: [...] The first line will print the currently active project if you're inside a -Scrapy project. In this example it was run from outside a project. If run from inside -a project it would have printed something like this:: +Scrapy project. In this example, it was run from outside a project. If run from +inside a project, it would have printed something like this:: Scrapy X.Y - project: myproject @@ -137,8 +137,8 @@ project:: scrapy startproject myproject [project_dir] -That will create a Scrapy project under the ``project_dir`` directory. -If ``project_dir`` wasn't specified, ``project_dir`` will be the same as ``myproject``. +That will create a Scrapy project under the ``project_dir`` directory. If +``project_dir`` isn't specified, it defaults to ``myproject``. Next, you go inside the new project directory:: @@ -159,7 +159,8 @@ For example, to create a new spider:: Some Scrapy commands (like :command:`crawl`) must be run from inside a Scrapy project. See the :ref:`commands reference ` below for more -information on which commands must be run from inside projects, and which not. +information on which commands must be run from inside projects and which ones +do not. Also keep in mind that some commands may have slightly different behaviours when running them from inside projects. For example, the fetch command will use @@ -186,7 +187,7 @@ And you can see all available commands with:: There are two kinds of commands, those that only work from inside a Scrapy project (Project-specific commands) and those that also work without an active Scrapy project (Global commands), though they may behave slightly differently -when run from inside a project (as they would use the project overridden +when run from inside a project (as they would use the project-overridden settings). Global commands: @@ -218,8 +219,7 @@ startproject * Requires project: *no* Creates a new Scrapy project named ``project_name``, under the ``project_dir`` -directory. -If ``project_dir`` wasn't specified, ``project_dir`` will be the same as ``project_name``. +directory. If ``project_dir`` isn't specified, it defaults to ``project_name``. Usage example:: @@ -233,7 +233,10 @@ genspider * Syntax: ``scrapy genspider [-t template] `` * Requires project: *no* -Creates a new spider in the current folder or in the current project's ``spiders`` folder, if called from inside a project. The ```` parameter is set as the spider's ``name``, while ```` is used to generate the ``allowed_domains`` and ``start_urls`` spider's attributes. +Creates a new spider in the current folder or in the current project's +``spiders`` folder, if called from inside a project. The ```` parameter +becomes the spider's ``name``, while ```` is used to generate +the spider's ``allowed_domains`` and ``start_urls`` attributes. Usage example:: @@ -250,9 +253,9 @@ Usage example:: $ scrapy genspider -t crawl scrapyorg scrapy.org Created spider 'scrapyorg' using template 'crawl' -This is just a convenient shortcut command for creating spiders based on -pre-defined templates, but certainly not the only way to create spiders. You -can just create the spider source code files yourself, instead of using this +This command is just a convenient shortcut for creating spiders based on +pre-defined templates, but it's certainly not the only way to create spiders; +you can create the spider source code files yourself instead of using this command. .. command:: crawl @@ -273,9 +276,13 @@ Supported options: * ``-a NAME=VALUE``: set a spider argument (may be repeated) -* ``--output FILE`` or ``-o FILE``: append scraped items to the end of FILE (use - for stdout). To define the output format, set a colon at the end of the output URI (i.e. ``-o FILE:FORMAT``) +* ``--output FILE`` or ``-o FILE``: append scraped items to the end of FILE + (use ``-`` for stdout). To define the output format, add a colon at the end + of the output URI (for example, ``-o FILE:FORMAT``) -* ``--overwrite-output FILE`` or ``-O FILE``: dump scraped items into FILE, overwriting any existing file. To define the output format, set a colon at the end of the output URI (i.e. ``-O FILE:FORMAT``) +* ``--overwrite-output FILE`` or ``-O FILE``: dump scraped items into FILE, + overwriting any existing file. To define the output format, add a colon at + the end of the output URI (for example, ``-O FILE:FORMAT``) Usage examples:: @@ -283,10 +290,10 @@ Usage examples:: [ ... myspider starts crawling ... ] $ scrapy crawl -o myfile:csv myspider - [ ... myspider starts crawling and appends the result to the file myfile in csv format ... ] + [ ... myspider starts crawling and appends the result to the file myfile in CSV format ... ] $ scrapy crawl -O myfile:json myspider - [ ... myspider starts crawling and saves the result in myfile in json format overwriting the original content... ] + [ ... myspider starts crawling and saves the result in myfile in JSON format, overwriting the original content ... ] .. command:: check @@ -362,7 +369,7 @@ Edit the given spider using the editor defined in the ``EDITOR`` environment variable or (if unset) the :setting:`EDITOR` setting. This command is provided only as a convenient shortcut for the most common -case, the developer is of course free to choose any tool or IDE to write and +case; the developer is of course free to choose any tool or IDE to write and debug spiders. Usage example:: @@ -386,7 +393,7 @@ attribute which overrides the User Agent, it will use that one. So this command can be used to "see" how your spider would fetch a certain page. -If used outside a project, no particular per-spider behaviour would be applied +If used outside a project, no particular per-spider behaviour will be applied, and it will just use the default Scrapy downloader settings. Supported options: @@ -448,10 +455,10 @@ shell * Syntax: ``scrapy shell [url]`` * Requires project: *no* -Starts the Scrapy shell for the given URL (if given) or empty if no URL is -given. Also supports UNIX-style local file paths, either relative with -``./`` or ``../`` prefixes or absolute file paths. -See :ref:`topics-shell` for more info. +Starts the Scrapy shell for the given URL (if provided) or leaves it empty if +no URL is given. It also supports UNIX-style local file paths, either relative +with ``./`` or ``../`` prefixes or absolute file paths. See :ref:`topics-shell` +for more info. Supported options: @@ -501,11 +508,13 @@ Supported options: * ``--callback`` or ``-c``: spider method to use as callback for parsing the response -* ``--meta`` or ``-m``: additional request meta that will be passed to the callback - request. This must be a valid json string. Example: --meta='{"foo" : "bar"}' +* ``--meta`` or ``-m``: additional request meta that will be passed to the + callback request. This must be a valid JSON string. Example: + ``--meta='{"foo": "bar"}'`` -* ``--cbkwargs``: additional keyword arguments that will be passed to the callback. - This must be a valid json string. Example: --cbkwargs='{"foo" : "bar"}' +* ``--cbkwargs``: additional keyword arguments that will be passed to the + callback. This must be a valid JSON string. Example: ``--cbkwargs='{"foo": + "bar"}'`` * ``--pipelines``: :ref:`process items through pipelines ` @@ -555,8 +564,8 @@ settings Get the value of a Scrapy setting. -If used inside a project it'll show the project setting value, otherwise it'll -show the default Scrapy value for that setting. +If used inside a project, it'll show the project setting value; otherwise, +it'll show the default Scrapy value for that setting. Example usage:: @@ -621,7 +630,7 @@ spider or a special internal one: * :command:`view` They use an internal instance of :class:`scrapy.crawler.AsyncCrawlerProcess` or -:class:`scrapy.crawler.CrawlerProcess` for this. In most cases this detail +:class:`scrapy.crawler.CrawlerProcess` for this. In most cases, this detail shouldn't matter to the user running the command, but when the user :ref:`needs a non-default Twisted reactor `, it may be important. @@ -640,9 +649,9 @@ project-level setting is set to :ref:`the asyncio reactor ` `) and :ref:`the setting of the spider being run ` is set to :ref:`a different one `, because :class:`~scrapy.crawler.AsyncCrawlerProcess` only supports the asyncio reactor. -In this case you should set the :setting:`FORCE_CRAWLER_PROCESS` setting to +In this case, you should set the :setting:`FORCE_CRAWLER_PROCESS` setting to ``True`` (at the project level or via the command line) so that Scrapy uses -:class:`~scrapy.crawler.CrawlerProcess` which supports all reactors. +:class:`~scrapy.crawler.CrawlerProcess`, which supports all reactors. Custom project commands ======================= @@ -679,7 +688,7 @@ You can also add Scrapy commands from an external library by adding a ``scrapy.commands`` section in the entry points of the library ``setup.py`` file. -The following example adds ``my_command`` command: +The following example adds the ``my_command`` command: .. skip: next diff --git a/docs/topics/components.rst b/docs/topics/components.rst index 354375577..d56f7794b 100644 --- a/docs/topics/components.rst +++ b/docs/topics/components.rst @@ -80,8 +80,8 @@ For example: print("log is enabled!") Components do not need to declare their custom settings programmatically. -However, they should document them, so that users know they exist and how to -use them. +However, they should document them so that users know they exist and how to use +them. It is a good practice to prefix custom settings with the name of the component, to avoid collisions with custom settings of other existing (or future) @@ -97,8 +97,8 @@ initialization, and if ``False``, raise :exc:`~scrapy.exceptions.NotConfigured`. When choosing a name for a custom setting, it is also a good idea to have a -look at the names of :ref:`built-in settings `, to try to -maintain consistency with them. +look at the names of :ref:`built-in settings ` to maintain +consistency with them. .. _enforce-component-requirements: @@ -118,7 +118,7 @@ In the case of :ref:`downloader middlewares `, `, and :ref:`spider middlewares `, you should raise :exc:`~scrapy.exceptions.NotConfigured`, passing a description of the issue as -a parameter to the exception so that it is printed in the logs, for the user to +a parameter to the exception so that it is printed in the logs for the user to see. For other components, feel free to raise whatever other exception feels right to you; for example, :exc:`RuntimeError` would make sense for a Scrapy version mismatch, while :exc:`ValueError` may be better if the issue is the @@ -139,7 +139,7 @@ If your requirement is a minimum Scrapy version, you may use if parse_version(scrapy.__version__) < parse_version("2.7"): raise RuntimeError( f"{MyComponent.__qualname__} requires Scrapy 2.7 or " - f"later, which allow defining the process_spider_output " + f"later, which allows defining the process_spider_output " f"method of spider middlewares as an asynchronous " f"generator." ) @@ -151,7 +151,7 @@ The following function can be used to create an instance of a component class: .. autofunction:: scrapy.utils.misc.build_from_crawler -The following function can also be useful when implementing a component, to +The following function can also be useful when implementing a component to report the import path of the component class, e.g. when reporting problems: .. autofunction:: scrapy.utils.python.global_object_name diff --git a/docs/topics/contracts.rst b/docs/topics/contracts.rst index df67bee02..eba2c91fa 100644 --- a/docs/topics/contracts.rst +++ b/docs/topics/contracts.rst @@ -1,16 +1,16 @@ .. _topics-contracts: -================= -Spiders Contracts -================= +================ +Spider Contracts +================ -Testing spiders can get particularly annoying and while nothing prevents you -from writing unit tests the task gets cumbersome quickly. Scrapy offers an +Testing spiders can get particularly annoying, and while nothing prevents you +from writing unit tests, the task gets cumbersome quickly. Scrapy offers an integrated way of testing your spiders by the means of contracts. -This allows you to test each callback of your spider by hardcoding a sample url -and check various constraints for how the callback processes the response. Each -contract is prefixed with an ``@`` and included in the docstring. See the +This allows you to test each callback of your spider by hardcoding a sample URL +and checking various constraints for how the callback processes the response. +Each contract is prefixed with an ``@`` and included in the docstring. See the following example: .. code-block:: python @@ -45,7 +45,7 @@ Use the :command:`check` command to run the contract checks. Custom Contracts ================ -If you find you need more power than the built-in Scrapy contracts you can +If you find you need more power than the built-in Scrapy contracts, you can create and load your own contracts in the project by using the :setting:`SPIDER_CONTRACTS` setting: @@ -68,7 +68,7 @@ override three methods: .. method:: pre_process(response) This allows hooking in various checks on the response received from the - sample request, before it's being passed to the callback. + sample request, before it is passed to the callback. .. method:: post_process(output) @@ -92,10 +92,8 @@ response received: class HasHeaderContract(Contract): - """ - Demo contract which checks the presence of a custom header - @has_header X-CustomHeader - """ + """Demo contract that checks the presence of a custom header: + @has_header X-CustomHeader""" name = "has_header" @@ -110,8 +108,8 @@ Detecting check runs ==================== When ``scrapy check`` is running, the ``SCRAPY_CHECK`` environment variable is -set to the ``true`` string. You can use :data:`os.environ` to perform any change to -your spiders or your settings when ``scrapy check`` is used: +set to the ``true`` string. You can use :data:`os.environ` to make any changes +to your spiders or your settings when ``scrapy check`` is used: .. code-block:: python diff --git a/docs/topics/coroutines.rst b/docs/topics/coroutines.rst index b7ddb0a57..0fd040f54 100644 --- a/docs/topics/coroutines.rst +++ b/docs/topics/coroutines.rst @@ -60,7 +60,7 @@ hence use coroutine syntax (e.g. ``await``, ``async for``, ``async with``): Using Deferred-based APIs ========================= -In addition to native coroutine APIs Scrapy has some APIs that return a +In addition to native coroutine APIs, Scrapy has some APIs that return a :class:`~twisted.internet.defer.Deferred` object or take a user-supplied function that returns a :class:`~twisted.internet.defer.Deferred` object. These APIs are also asynchronous but don't yet support native ``async def`` syntax. @@ -121,7 +121,7 @@ return coroutines are listed in :ref:`coroutine-support`): - ``stat_file()`` -In most cases you can use these APIs in code that otherwise uses coroutines, by +In most cases, you can use these APIs in code that otherwise uses coroutines by wrapping a :class:`~twisted.internet.defer.Deferred` object into a :class:`~asyncio.Future` object or vice versa. See :ref:`asyncio-await-dfd` for more information about this. @@ -174,8 +174,8 @@ becomes: Coroutines may be used to call asynchronous code. This includes other coroutines, functions that return Deferreds and functions that return -:term:`awaitable objects ` such as :class:`~asyncio.Future`. -This means you can use many useful Python libraries providing such code: +:term:`awaitable objects ` such as :class:`~asyncio.Future`. This +means you can use many useful Python libraries that provide such code: .. skip: next .. code-block:: python @@ -197,8 +197,8 @@ This means you can use many useful Python libraries providing such code: # ... use response and additional_data to yield items and requests .. note:: Many libraries that use coroutines, such as `aio-libs`_, require the - :mod:`asyncio` loop and to use them you need to - :doc:`enable asyncio support in Scrapy`. + :mod:`asyncio` loop, and to use them you need to :doc:`enable asyncio + support in Scrapy`. .. note:: If you want to ``await`` on Deferreds while using the asyncio reactor, you need to :ref:`wrap them`. diff --git a/docs/topics/debug.rst b/docs/topics/debug.rst index 988e37bbd..d45e33f0e 100644 --- a/docs/topics/debug.rst +++ b/docs/topics/debug.rst @@ -5,7 +5,7 @@ Debugging Spiders ================= This document explains the most common techniques for debugging spiders. -Consider the following Scrapy spider below: +Consider the following Scrapy spider: .. skip: next .. code-block:: python @@ -40,19 +40,19 @@ Consider the following Scrapy spider below: # populate more `item` fields return item -Basically this is a simple spider which parses two pages of items (the -start_urls). Items also have a details page with additional information, so we -use the ``cb_kwargs`` functionality of :class:`~scrapy.Request` to pass a -partially populated item. +Basically, this is a simple spider that parses two pages of items (the start +URLs). Items also have a details page with additional information, so we use +the ``cb_kwargs`` functionality of :class:`~scrapy.Request` to pass a partially +populated item. Parse Command ============= The most basic way of checking the output of your spider is to use the -:command:`parse` command. It allows to check the behaviour of different parts -of the spider at the method level. It has the advantage of being flexible and -simple to use, but does not allow debugging code inside a method. +:command:`parse` command. It allows you to check the behaviour of different +parts of the spider at the method level. It has the advantage of being flexible +and simple to use, but it does not allow debugging code inside a method. .. highlight:: none @@ -90,7 +90,7 @@ Using the ``--verbose`` or ``-v`` option we can see the status at each depth lev # Requests ----------------------------------------------------------------- [] -Checking items scraped from a single start_url, can also be easily achieved +Checking items scraped from a single start URL can also be easily achieved using:: $ scrapy parse --spider=myspider -d 3 'http://example.com/page1' @@ -101,10 +101,10 @@ using:: Scrapy Shell ============ -While the :command:`parse` command is very useful for checking behaviour of a -spider, it is of little help to check what happens inside a callback, besides -showing the response received and the output. How to debug the situation when -``parse_details`` sometimes receives no item? +While the :command:`parse` command is very useful for checking the behaviour of +a spider, it is of little help when checking what happens inside a callback +besides showing the response received and the output. How do you debug the +situation when ``parse_details`` sometimes receives no item? .. highlight:: python @@ -129,7 +129,7 @@ See also: :ref:`topics-shell-inspect-response`. Open in browser =============== -Sometimes you just want to see how a certain response looks in a browser, you +Sometimes you just want to see how a certain response looks in a browser; you can use the :func:`~scrapy.utils.response.open_in_browser` function for that: .. autofunction:: scrapy.utils.response.open_in_browser @@ -140,7 +140,7 @@ Logging Logging is another useful option for getting information about your spider run. Although not as convenient, it comes with the advantage that the logs will be -available in all future runs should they be necessary again: +available in all future runs should you need them again: .. code-block:: python diff --git a/docs/topics/deploy.rst b/docs/topics/deploy.rst index f3515b4be..7217fc6a3 100644 --- a/docs/topics/deploy.rst +++ b/docs/topics/deploy.rst @@ -24,8 +24,8 @@ Deploying to a Scrapyd Server `Scrapyd`_ is an open source application to run Scrapy spiders. It provides a server with HTTP API, capable of running and monitoring Scrapy spiders. -To deploy spiders to Scrapyd, you can use the scrapyd-deploy tool provided by -the `scrapyd-client`_ package. Please refer to the `scrapyd-deploy +To deploy spiders to Scrapyd, you can use the ``scrapyd-deploy`` tool provided +by the `scrapyd-client`_ package. Please refer to the `scrapyd-deploy documentation`_ for more information. Scrapyd is maintained by some of the Scrapy developers. @@ -38,12 +38,12 @@ Deploying to Zyte Scrapy Cloud `Zyte Scrapy Cloud`_ is a hosted, cloud-based service by Zyte_, the company behind Scrapy. -Zyte Scrapy Cloud removes the need to setup and monitor servers and provides a +Zyte Scrapy Cloud removes the need to set up and monitor servers and provides a nice UI to manage spiders and review scraped items, logs and stats. -To deploy spiders to Zyte Scrapy Cloud you can use the `shub`_ command line -tool. -Please refer to the `Zyte Scrapy Cloud documentation`_ for more information. +To deploy spiders to Zyte Scrapy Cloud, you can use the ``shub`` command-line +tool. Please refer to the `Zyte Scrapy Cloud documentation`_ for more +information. Zyte Scrapy Cloud is compatible with Scrapyd and one can switch between them as needed - the configuration is read from the ``scrapy.cfg`` file diff --git a/docs/topics/developer-tools.rst b/docs/topics/developer-tools.rst index a5ffe00f1..31ecb7089 100644 --- a/docs/topics/developer-tools.rst +++ b/docs/topics/developer-tools.rst @@ -4,10 +4,10 @@ Using your browser's Developer Tools for scraping ================================================= -Here is a general guide on how to use your browser's Developer Tools -to ease the scraping process. Today almost all browsers come with -built in `Developer Tools`_ and although we will use Firefox in this -guide, the concepts are applicable to any other browser. +Here is a general guide on how to use your browser's Developer Tools to ease +the scraping process. Today almost all browsers come with built-in `Developer +Tools`_, and although we will use Firefox in this guide, the concepts are +applicable to any other browser. In this guide we'll introduce the basic tools to use from a browser's Developer Tools by scraping `quotes.toscrape.com`_. @@ -19,9 +19,9 @@ Caveats with inspecting the live browser DOM Since Developer Tools operate on a live browser DOM, what you'll actually see when inspecting the page source is not the original HTML, but a modified one -after applying some browser clean up and executing JavaScript code. Firefox, -in particular, is known for adding ```` elements to tables. Scrapy, on -the other hand, does not modify the original page HTML, so you won't be able to +after applying some browser clean up and executing JavaScript code. Firefox, in +particular, is known for adding ```` elements to tables. Scrapy, on the +other hand, does not modify the original page HTML, so you won't be able to extract any data if you use ```` in your XPath expressions. Therefore, you should keep in mind the following things: @@ -42,17 +42,16 @@ Inspecting a website ==================== By far the most handy feature of the Developer Tools is the `Inspector` -feature, which allows you to inspect the underlying HTML code of -any webpage. To demonstrate the Inspector, let's look at the -`quotes.toscrape.com`_-site. +feature, which allows you to inspect the underlying HTML code of any webpage. +To demonstrate the Inspector, let's look at the `quotes.toscrape.com`_ site. On the site we have a total of ten quotes from various authors with specific tags, as well as the Top Ten Tags. Let's say we want to extract all the quotes on this page, without any meta-information about authors, tags, etc. -Instead of viewing the whole source code for the page, we can simply right click -on a quote and select ``Inspect Element (Q)``, which opens up the `Inspector`. -In it you should see something like this: +Instead of viewing the whole source code for the page, we can simply +right-click on a quote and select ``Inspect Element (Q)``, which opens up the +`Inspector`. In it you should see something like this: .. image:: _images/inspector_01.png :width: 777 @@ -76,10 +75,10 @@ anywhere. The advantage of the `Inspector` is that it automatically expands and collapses sections and tags of a webpage, which greatly improves readability. You can -expand and collapse a tag by clicking on the arrow in front of it or by double -clicking directly on the tag. If we expand the ``span`` tag with the ``class= -"text"`` we will see the quote-text we clicked on. The `Inspector` lets you -copy XPaths to selected elements. Let's try it out. +expand and collapse a tag by clicking on the arrow in front of it or by +double-clicking directly on the tag. If we expand the ``span`` tag with the +``class="text"`` attribute, we see the quote text we clicked on. The +`Inspector` lets you copy XPaths to selected elements. Let's try it out. First open the Scrapy shell at https://quotes.toscrape.com/ in a terminal: @@ -99,17 +98,17 @@ Then, back to your web browser, right-click on the ``span`` tag, select >>> response.xpath("/html/body/div/div[2]/div[1]/div[1]/span[1]/text()").getall() ['“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”'] -Adding ``text()`` at the end we are able to extract the first quote with this -basic selector. But this XPath is not really that clever. All it does is -go down a desired path in the source code starting from ``html``. So let's -see if we can refine our XPath a bit: +Adding ``text()`` at the end, we are able to extract the first quote with this +basic selector. But this XPath is not really that clever. All it does is go +down a desired path in the source code starting from ``html``. So let's see if +we can refine our XPath a bit: -If we check the `Inspector` again we'll see that directly beneath our -expanded ``div`` tag we have nine identical ``div`` tags, each with the -same attributes as our first. If we expand any of them, we'll see the same -structure as with our first quote: Two ``span`` tags and one ``div`` tag. We can -expand each ``span`` tag with the ``class="text"`` inside our ``div`` tags and -see each quote: +If we check the `Inspector` again we'll see that directly beneath our expanded +``div`` tag we have nine identical ``div`` tags, each with the same attributes +as our first. If we expand any of them, we'll see the same structure as with +our first quote: two ``span`` tags and one ``div`` tag. We can expand each +``span`` tag with the ``class="text"`` inside our ``div`` tags and see each +quote: .. code-block:: html @@ -145,14 +144,14 @@ source code or directly scrolling to an element you selected. Let's demonstrate a use case: Say you want to find the ``Next`` button on the page. Type ``Next`` into the -search bar on the top right of the `Inspector`. You should get two results. -The first is a ``li`` tag with the ``class="next"``, the second the text -of an ``a`` tag. Right click on the ``a`` tag and select ``Scroll into View``. -If you hover over the tag, you'll see the button highlighted. From here -we could easily create a :ref:`Link Extractor ` to -follow the pagination. On a simple site such as this, there may not be -the need to find an element visually but the ``Scroll into View`` function -can be quite useful on complex sites. +search bar on the top-right of the `Inspector`. You should get two results. The +first is a ``li`` tag with the ``class="next"``, the second the text of an +``a`` tag. Right-click on the ``a`` tag and select ``Scroll into View``. If you +hover over the tag, you'll see the button highlighted. From here we could +easily create a :ref:`Link Extractor ` to follow the +pagination. On a simple site such as this, there may not be the need to find an +element visually but the ``Scroll into View`` function can be quite useful on +complex sites. Note that the search bar can also be used to search for and test CSS selectors. For example, you could search for ``span.text`` to find @@ -161,19 +160,19 @@ exactly the ``span`` tag with the ``class="text"`` in the page. .. _topics-network-tool: -The Network-tool +The Network tool ================ -While scraping you may come across dynamic webpages where some parts -of the page are loaded dynamically through multiple requests. While -this can be quite tricky, the `Network`-tool in the Developer Tools -greatly facilitates this task. To demonstrate the Network-tool, let's -take a look at the page `quotes.toscrape.com/scroll`_. +While scraping you may come across dynamic webpages where some parts of the +page are loaded dynamically through multiple requests. While this can be quite +tricky, the `Network` tool in the Developer Tools greatly facilitates this +task. To demonstrate the Network tool, let's take a look at the page +`quotes.toscrape.com/scroll`_. -The page is quite similar to the basic `quotes.toscrape.com`_-page, -but instead of the above-mentioned ``Next`` button, the page -automatically loads new quotes when you scroll to the bottom. We -could go ahead and try out different XPaths directly, but instead -we'll check another quite useful command from the Scrapy shell: +The page is quite similar to the basic `quotes.toscrape.com`_ page, but instead +of the above-mentioned ``Next`` button, the page automatically loads new quotes +when you scroll to the bottom. We could go ahead and try out different XPaths +directly, but instead we'll check another quite useful command from the Scrapy +shell: .. skip: next @@ -192,12 +191,11 @@ bar with the word ``Loading...``. :height: 296 :alt: Response from quotes.toscrape.com/scroll -The ``view(response)`` command let's us view the response our -shell or later our spider receives from the server. Here we see -that some basic template is loaded which includes the title, -the login-button and the footer, but the quotes are missing. This -tells us that the quotes are being loaded from a different request -than ``quotes.toscrape/scroll``. +The ``view(response)`` command lets us view the response our shell or later our +spider receives from the server. Here we see that some basic template is loaded +which includes the title, the login button and the footer, but the quotes are +missing. This tells us that the quotes are being loaded from a different +request than ``quotes.toscrape.com/scroll``. If you click on the ``Network`` tab, you will probably only see two entries. The first thing we do is enable persistent logs by @@ -218,30 +216,29 @@ Here we see every request that has been made when reloading the page and can inspect each request and its response. So let's find out where our quotes are coming from: -First click on the request with the name ``scroll``. On the right -you can now inspect the request. In ``Headers`` you'll find details -about the request headers, such as the URL, the method, the IP-address, -and so on. We'll ignore the other tabs and click directly on ``Response``. +First click on the request with the name ``scroll``. On the right you can now +inspect the request. In ``Headers`` you'll find details about the request +headers, such as the URL, the method, the IP address, and so on. We'll ignore +the other tabs and click directly on ``Response``. -What you should see in the ``Preview`` pane is the rendered HTML-code, -that is exactly what we saw when we called ``view(response)`` in the -shell. Accordingly the ``type`` of the request in the log is ``html``. -The other requests have types like ``css`` or ``js``, but what -interests us is the one request called ``quotes?page=1`` with the -type ``json``. +What you should see in the ``Preview`` pane is the rendered HTML code, exactly +what we saw when we called ``view(response)`` in the shell. Accordingly, the +``type`` of the request in the log is ``html``. The other requests have types +like ``css`` or ``js``, but what interests us is the one request called +``quotes?page=1`` with the type ``json``. If we click on this request, we see that the request URL is -``https://quotes.toscrape.com/api/quotes?page=1`` and the response -is a JSON-object that contains our quotes. We can also right-click -on the request and open ``Open in new tab`` to get a better overview. +``https://quotes.toscrape.com/api/quotes?page=1`` and the response is a JSON +object that contains our quotes. We can also right-click on the request and +open ``Open in new tab`` to get a better overview. .. image:: _images/network_03.png :width: 777 :height: 375 - :alt: JSON-object returned from the quotes.toscrape API + :alt: JSON object returned from the quotes.toscrape API -With this response we can now easily parse the JSON-object and -also request each page to get every quote on the site: +With this response we can now easily parse the JSON object and also request +each page to get every quote on the site: .. code-block:: python @@ -263,15 +260,14 @@ also request each page to get every quote on the site: url = f"https://quotes.toscrape.com/api/quotes?page={self.page}" yield scrapy.Request(url=url, callback=self.parse) -This spider starts at the first page of the quotes-API. With each -response, we parse the ``response.text`` and assign it to ``data``. -This lets us operate on the JSON-object like on a Python dictionary. -We iterate through the ``quotes`` and print out the ``quote["text"]``. -If the handy ``has_next`` element is ``true`` (try loading -`quotes.toscrape.com/api/quotes?page=10`_ in your browser or a -page-number greater than 10), we increment the ``page`` attribute -and ``yield`` a new request, inserting the incremented page-number -into our ``url``. +This spider starts at the first page of the quotes-API. With each response, we +parse the ``response.text`` and assign it to ``data``. This lets us operate on +the JSON object like on a Python dictionary. We iterate through the ``quotes`` +and print out the ``quote["text"]``. If the handy ``has_next`` element is +``true`` (try loading `quotes.toscrape.com/api/quotes?page=10`_ in your browser +or a page number greater than 10), we increment the ``page`` attribute and +``yield`` a new request, inserting the incremented page number into our +``url``. .. _requests-from-curl: @@ -305,11 +301,11 @@ function to get a dictionary with the equivalent arguments: Note that to translate a cURL command into a Scrapy request, you may use `curl2scrapy `_. -As you can see, with a few inspections in the `Network`-tool we -were able to easily replicate the dynamic requests of the scrolling -functionality of the page. Crawling dynamic pages can be quite -daunting and pages can be very complex, but it (mostly) boils down -to identifying the correct request and replicating it in your spider. +As you can see, with a few inspections in the `Network` tool we were able to +easily replicate the dynamic requests of the scrolling functionality of the +page. Crawling dynamic pages can be quite daunting and pages can be very +complex, but it (mostly) boils down to identifying the correct request and +replicating it in your spider. .. _Developer Tools: https://en.wikipedia.org/wiki/Web_development_tools .. _quotes.toscrape.com: https://quotes.toscrape.com diff --git a/docs/topics/dynamic-content.rst b/docs/topics/dynamic-content.rst index 30b6536c7..31ab4d6a6 100644 --- a/docs/topics/dynamic-content.rst +++ b/docs/topics/dynamic-content.rst @@ -85,16 +85,16 @@ It might be enough to yield a :class:`~scrapy.Request` with the same HTTP method and URL. However, you may also need to reproduce the body, headers and form parameters (see :ref:`form`) of that request. -As all major browsers allow to export the requests in curl_ format, Scrapy -incorporates the method :meth:`~scrapy.Request.from_curl` to generate an equivalent -:class:`~scrapy.Request` from a cURL command. To get more information -visit :ref:`request from curl ` inside the network -tool section. +As all major browsers allow you to export requests in curl_ format, Scrapy +incorporates the method :meth:`~scrapy.Request.from_curl` to generate an +equivalent :class:`~scrapy.Request` from a cURL command. To get more +information visit :ref:`request from curl ` inside the +network tool section. Once you get the expected response, you can :ref:`extract the desired data from it `. -You can reproduce any request with Scrapy. However, some times reproducing all +You can reproduce any request with Scrapy. However, sometimes reproducing all necessary requests may not seem efficient in developer time. If that is your case, and crawling speed is not a major concern for you, you can alternatively consider :ref:`using a headless browser `. @@ -271,8 +271,8 @@ The following is a simple snippet to illustrate its usage within a Scrapy spider However, using `playwright-python`_ directly as in the above example -circumvents most of the Scrapy components (middlewares, dupefilter, etc). -We recommend using `scrapy-playwright`_ for a better integration. +circumvents most of the Scrapy components (middlewares, dupefilter, etc.). We +recommend using `scrapy-playwright`_ for a better integration. .. _CSS: https://en.wikipedia.org/wiki/Cascading_Style_Sheets .. _chompjs: https://github.com/Nykakin/chompjs diff --git a/docs/topics/exporters.rst b/docs/topics/exporters.rst index 56b995e18..7e0de3ff0 100644 --- a/docs/topics/exporters.rst +++ b/docs/topics/exporters.rst @@ -11,34 +11,34 @@ Once you have scraped your items, you often want to persist or export those items, to use the data in some other application. That is, after all, the whole purpose of the scraping process. -For this purpose Scrapy provides a collection of Item Exporters for different +For this purpose, Scrapy provides a collection of Item Exporters for different output formats, such as XML, CSV or JSON. Using Item Exporters ==================== If you are in a hurry, and just want to use an Item Exporter to output scraped -data see the :ref:`topics-feed-exports`. Otherwise, if you want to know how +data, see the :ref:`topics-feed-exports`. Otherwise, if you want to know how Item Exporters work or need more custom functionality (not covered by the default exports), continue reading below. -In order to use an Item Exporter, you must instantiate it with its required -args. Each Item Exporter requires different arguments, so check each exporter -documentation to be sure, in :ref:`topics-exporters-reference`. After you have +In order to use an Item Exporter, you must instantiate it with its required +args. Each Item Exporter requires different arguments, so check each exporter's +documentation in :ref:`topics-exporters-reference` to be sure. After you have instantiated your exporter, you have to: -1. call the method :meth:`~BaseItemExporter.start_exporting` in order to -signal the beginning of the exporting process +1. Call the method :meth:`~BaseItemExporter.start_exporting` in order to + signal the beginning of the exporting process -2. call the :meth:`~BaseItemExporter.export_item` method for each item you want -to export +2. Call the :meth:`~BaseItemExporter.export_item` method for each item you want + to export -3. and finally call the :meth:`~BaseItemExporter.finish_exporting` to signal -the end of the exporting process +3. Finally, call the :meth:`~BaseItemExporter.finish_exporting` method to + signal the end of the exporting process -Here you can see an :doc:`Item Pipeline ` which uses multiple -Item Exporters to group scraped items to different files according to the -value of one of their fields: +Here you can see an :doc:`Item Pipeline ` that uses multiple +Item Exporters to distribute scraped items into different files according to +the value of one of their fields: .. code-block:: python @@ -224,9 +224,10 @@ BaseItemExporter defining what fields to export, whether to export empty fields, or which encoding to use. - These features can be configured through the ``__init__`` method arguments which - populate their respective instance attributes: :attr:`fields_to_export`, - :attr:`export_empty_fields`, :attr:`encoding`, :attr:`indent`. + These features can be configured through the ``__init__`` method arguments, + which populate their respective instance attributes: + :attr:`fields_to_export`, :attr:`export_empty_fields`, :attr:`encoding`, + :attr:`indent`. .. method:: export_item(item) @@ -341,8 +342,9 @@ XmlItemExporter Exports items in XML format to the specified file object. - :param file: the file-like object to use for exporting the data. Its ``write`` method should - accept ``bytes`` (a disk file opened in binary mode, a ``io.BytesIO`` object, etc) + :param file: the file-like object to use for exporting the data. Its + ``write`` method should accept ``bytes`` (a disk file opened in binary + mode, an ``io.BytesIO`` object, etc) :param root_element: The name of root element in the exported XML. :type root_element: str @@ -381,7 +383,7 @@ XmlItemExporter Item(name=["John", "Doe"], age="23") - Would be serialized as: + It would be serialized as: .. code-block:: xml @@ -406,8 +408,9 @@ CsvItemExporter CSV columns, their order and their column names. The :attr:`export_empty_fields` attribute has no effect on this exporter. - :param file: the file-like object to use for exporting the data. Its ``write`` method should - accept ``bytes`` (a disk file opened in binary mode, a ``io.BytesIO`` object, etc) + :param file: the file-like object to use for exporting the data. Its + ``write`` method should accept ``bytes`` (a disk file opened in binary + mode, an ``io.BytesIO`` object, etc) :param include_headers_line: If enabled, makes the exporter output a header line with the field names taken from @@ -441,8 +444,9 @@ PickleItemExporter Exports items in pickle format to the given file-like object. - :param file: the file-like object to use for exporting the data. Its ``write`` method should - accept ``bytes`` (a disk file opened in binary mode, a ``io.BytesIO`` object, etc) + :param file: the file-like object to use for exporting the data. Its + ``write`` method should accept ``bytes`` (a disk file opened in binary + mode, an ``io.BytesIO`` object, etc) :param protocol: The pickle protocol to use. :type protocol: int @@ -452,7 +456,7 @@ PickleItemExporter The additional keyword arguments of this ``__init__`` method are passed to the :class:`BaseItemExporter` ``__init__`` method. - Pickle isn't a human readable format, so no output examples are provided. + Pickle isn't a human-readable format, so no output examples are provided. PprintItemExporter ------------------ @@ -461,8 +465,9 @@ PprintItemExporter Exports items in pretty print format to the specified file object. - :param file: the file-like object to use for exporting the data. Its ``write`` method should - accept ``bytes`` (a disk file opened in binary mode, a ``io.BytesIO`` object, etc) + :param file: the file-like object to use for exporting the data. Its + ``write`` method should accept ``bytes`` (a disk file opened in binary + mode, an ``io.BytesIO`` object, etc) The additional keyword arguments of this ``__init__`` method are passed to the :class:`BaseItemExporter` ``__init__`` method. @@ -487,8 +492,9 @@ JsonItemExporter arguments to the :class:`~json.JSONEncoder` ``__init__`` method, so you can use any :class:`~json.JSONEncoder` ``__init__`` method argument to customize this exporter. - :param file: the file-like object to use for exporting the data. Its ``write`` method should - accept ``bytes`` (a disk file opened in binary mode, a ``io.BytesIO`` object, etc) + :param file: the file-like object to use for exporting the data. Its + ``write`` method should accept ``bytes`` (a disk file opened in binary + mode, an ``io.BytesIO`` object, etc) A typical output of this exporter would be: @@ -499,10 +505,10 @@ JsonItemExporter .. _json-with-large-data: - .. warning:: JSON is very simple and flexible serialization format, but it - doesn't scale well for large amounts of data since incremental (aka. + .. warning:: JSON is a very simple and flexible serialization format, but it + doesn't scale well for large amounts of data because incremental (aka stream-mode) parsing is not well supported (if at all) among JSON parsers - (on any language), and most of them just parse the entire object in + in any language, and most of them just parse the entire object in memory. If you want the power and simplicity of JSON with a more stream-friendly format, consider using :class:`JsonLinesItemExporter` instead, or splitting the output in multiple chunks. @@ -518,8 +524,9 @@ JsonLinesItemExporter the :class:`~json.JSONEncoder` ``__init__`` method, so you can use any :class:`~json.JSONEncoder` ``__init__`` method argument to customize this exporter. - :param file: the file-like object to use for exporting the data. Its ``write`` method should - accept ``bytes`` (a disk file opened in binary mode, a ``io.BytesIO`` object, etc) + :param file: the file-like object to use for exporting the data. Its + ``write`` method should accept ``bytes`` (a disk file opened in binary + mode, an ``io.BytesIO`` object, etc) A typical output of this exporter would be: diff --git a/docs/topics/extensions.rst b/docs/topics/extensions.rst index 6d61cc342..b14b7c934 100644 --- a/docs/topics/extensions.rst +++ b/docs/topics/extensions.rst @@ -48,8 +48,8 @@ tasks triggered by them. Sample extension ---------------- -Here we will implement a simple extension to illustrate the concepts described -in the previous section. This extension will log a message every time: +Here we implement a simple extension to illustrate the concepts described in +the previous section. This extension will log a message every time: * a spider is opened * a spider is closed @@ -58,7 +58,7 @@ in the previous section. This extension will log a message every time: The extension will be enabled through the ``MYEXT_ENABLED`` setting and the number of items will be specified through the ``MYEXT_ITEMCOUNT`` setting. -Here is the code of such extension: +Here is the code for that extension: .. code-block:: python @@ -163,7 +163,7 @@ Provides a telnet console for getting into a Python interpreter inside the currently running Scrapy process, which can be very useful for debugging. The telnet console must be enabled by the :setting:`TELNETCONSOLE_ENABLED` -setting, and the server will listen in the port specified in +setting, and the server will listen on the port specified in :setting:`TELNETCONSOLE_PORT`. .. _topics-extensions-ref-memusage: @@ -203,7 +203,8 @@ Memory debugger extension An extension for debugging memory usage. It collects information about: * objects uncollected by the Python garbage collector -* objects left alive that shouldn't. For more info, see :ref:`topics-leaks-trackrefs` +* objects left alive that shouldn't be. For more info, see + :ref:`topics-leaks-trackrefs` To enable this extension, turn on the :setting:`MEMDEBUG_ENABLED` setting. The info will be stored in the :stat:`memdebug/gc_garbage_count` and @@ -258,9 +259,9 @@ settings: .. note:: - When a certain closing condition is met, requests which are - currently in the downloader queue (up to :setting:`CONCURRENT_REQUESTS` - requests) are still processed. + When a certain closing condition is met, requests that are currently in the + downloader queue (up to :setting:`CONCURRENT_REQUESTS` + requests) are still processed. .. setting:: CLOSESPIDER_TIMEOUT @@ -270,7 +271,7 @@ CLOSESPIDER_TIMEOUT Default: ``0.0`` If the spider remains open for more than this number of seconds, it will be -automatically closed with the reason ``closespider_timeout``. If zero (or non +automatically closed with the reason ``closespider_timeout``. If zero (or not set), spiders won't be closed by timeout. .. setting:: CLOSESPIDER_TIMEOUT_NO_ITEM @@ -282,8 +283,8 @@ Default: ``0`` An integer which specifies a number of seconds. If the spider has not produced any items in the last number of seconds, it will be closed with the reason -``closespider_timeout_no_item``. If zero (or non set), spiders won't be closed -regardless if it hasn't produced any items. +``closespider_timeout_no_item``. If zero (or not set), spiders won't be closed +even if they have not produced any items. .. setting:: CLOSESPIDER_ITEMCOUNT @@ -293,9 +294,9 @@ CLOSESPIDER_ITEMCOUNT Default: ``0`` An integer which specifies a number of items. If the spider scrapes more than -that amount and those items are passed by the item pipeline, the -spider will be closed with the reason ``closespider_itemcount``. -If zero (or non set), spiders won't be closed by number of passed items. +that amount and those items are passed by the item pipeline, the spider will be +closed with the reason ``closespider_itemcount``. If zero (or not set), spiders +won't be closed by the number of passed items. .. setting:: CLOSESPIDER_PAGECOUNT @@ -306,7 +307,7 @@ Default: ``0`` An integer which specifies the maximum number of responses to crawl. If the spider crawls more than that, the spider will be closed with the reason -``closespider_pagecount``. If zero (or non set), spiders won't be closed by +``closespider_pagecount``. If zero (or not set), spiders won't be closed by the number of crawled responses. .. setting:: CLOSESPIDER_PAGECOUNT_NO_ITEM @@ -320,7 +321,7 @@ An integer which specifies the maximum number of consecutive responses to crawl without items scraped. If the spider crawls more consecutive responses than that and no items are scraped in the meantime, the spider will be closed with the reason ``closespider_pagecount_no_item``. If zero (or not set), spiders won't be -closed by number of crawled responses with no items. +closed by the number of crawled responses with no items. .. setting:: CLOSESPIDER_ERRORCOUNT @@ -331,8 +332,8 @@ Default: ``0`` An integer which specifies the maximum number of errors to receive before closing the spider. If the spider generates more than that number of errors, -it will be closed with the reason ``closespider_errorcount``. If zero (or non -set), spiders won't be closed by number of errors. +it will be closed with the reason ``closespider_errorcount``. If zero (or not +set), spiders won't be closed by the number of errors. .. module:: scrapy.extensions.periodic_log :synopsis: Periodic stats logging diff --git a/docs/topics/feed-exports.rst b/docs/topics/feed-exports.rst index 467abc989..f7ee3b941 100644 --- a/docs/topics/feed-exports.rst +++ b/docs/topics/feed-exports.rst @@ -5,9 +5,9 @@ Feed exports ============ One of the most frequently required features when implementing scrapers is -being able to store the scraped data properly and, quite often, that means -generating an "export file" with the scraped data (commonly called "export -feed") to be consumed by other systems. +properly storing the scraped data and, quite often, that means generating an +"export file" with the scraped data (commonly called an "export feed") to be +consumed by other systems. Scrapy provides this functionality out of the box with the Feed Exports, which allows you to generate feeds with the scraped items, using multiple @@ -31,7 +31,7 @@ For serializing the scraped data, the feed exports use the :ref:`Item exporters - :ref:`topics-feed-format-csv` - :ref:`topics-feed-format-xml` -But you can also extend the supported format through the +But you can also extend the supported formats through the :setting:`FEED_EXPORTERS` setting. .. _topics-feed-format-json: @@ -97,9 +97,9 @@ Marshal Storages ======== -When using the feed exports you define where to store the feed using one or multiple URIs_ -(through the :setting:`FEEDS` setting). The feed exports supports multiple -storage backend types which are defined by the URI scheme. +When using the feed exports you define where to store the feed using one or +multiple URIs_ (through the :setting:`FEEDS` setting). The feed exports support +multiple storage backend types that are defined by the URI scheme. The storages backends supported out of the box are: @@ -125,8 +125,8 @@ being created. These parameters are: - ``%(time)s`` - gets replaced by a timestamp when the feed is being created - ``%(name)s`` - gets replaced by the spider name -Any other named parameter gets replaced by the spider attribute of the same -name. For example, ``%(site_id)s`` would get replaced by the ``spider.site_id`` +Any other named parameter is replaced by the spider attribute of the same name. +For example, ``%(site_id)s`` would be replaced by the ``spider.site_id`` attribute the moment the feed is being created. Here are some examples to illustrate: @@ -164,9 +164,9 @@ The feeds are stored in the local filesystem. - Example URI: ``file:///tmp/export.csv`` - Required external libraries: none -Note that for the local filesystem storage (only) you can omit the scheme if -you specify a path (e.g. ``/tmp/export.csv``). -Alternatively you can also use a :class:`pathlib.Path` object. +Note that for the local filesystem storage you can omit the scheme if you +specify a path (e.g. ``/tmp/export.csv``). Alternatively, you can use a +:class:`pathlib.Path` object. .. _topics-feed-storage-ftp: .. _feed-storage-ftp: @@ -174,7 +174,7 @@ Alternatively you can also use a :class:`pathlib.Path` object. FTP --- -The feeds are stored in a FTP server. +The feeds are stored on an FTP server. - URI scheme: ``ftp`` - Example URI: ``ftp://user:pass@ftp.example.com/path/to/export.csv`` diff --git a/docs/topics/item-pipeline.rst b/docs/topics/item-pipeline.rst index a6aac78ac..2f830cedd 100644 --- a/docs/topics/item-pipeline.rst +++ b/docs/topics/item-pipeline.rst @@ -7,10 +7,10 @@ Item Pipeline After an item has been scraped by a spider, it is sent to the Item Pipeline which processes it through several components that are executed sequentially. -Each item pipeline component (sometimes referred as just "Item Pipeline") is a -Python class that implements a simple method. They receive an item and perform -an action over it, also deciding if the item should continue through the -pipeline or be dropped and no longer processed. +Each item pipeline component (sometimes referred to as just "Item Pipeline") is +a Python class that implements a simple method. Each component receives an item +and performs an action on it while deciding whether the item should continue +through the pipeline or be dropped and no longer processed. Typical uses of item pipelines are: @@ -28,7 +28,7 @@ implement the following method: .. method:: process_item(self, item) - This method is called for every item pipeline component. + Scrapy calls this method for every item processed by the pipeline component. `item` is an :ref:`item object `, see :ref:`supporting-item-types`. @@ -41,7 +41,7 @@ implement the following method: :param item: the scraped item :type item: :ref:`item object ` -Additionally, they may also implement the following methods: +Additionally, a component may implement the following methods: .. method:: open_spider(self) @@ -72,8 +72,8 @@ Price validation and dropping items with no prices Let's take a look at the following hypothetical pipeline that adjusts the ``price`` attribute for those items that do not include VAT -(``price_excludes_vat`` attribute), and drops those items which don't -contain a price: +(``price_excludes_vat`` attribute), and drops those items that don't contain a +price: .. code-block:: python @@ -120,16 +120,16 @@ format: self.file.write(line) return item -.. note:: The purpose of JsonWriterPipeline is just to introduce how to write - item pipelines. If you really want to store all scraped items into a JSON - file you should use the :ref:`Feed exports `. +.. note:: The JsonWriterPipeline example simply introduces how to write item + pipelines. If you really want to store all scraped items in a JSON file, + you should use the :ref:`Feed exports `. Write items to MongoDB ---------------------- -In this example we'll write items to MongoDB_ using pymongo_. -MongoDB address and database name are specified in Scrapy settings; -MongoDB collection is specified in a class attribute. +In this example, we'll write items to MongoDB_ using pymongo_. The MongoDB +address and database name are specified in the Scrapy settings; the MongoDB +collection is specified in a class attribute. The main point of this example is to show how to :ref:`get the crawler ` and how to clean up the resources properly. @@ -172,13 +172,13 @@ The main point of this example is to show how to :ref:`get the crawler .. _ScreenshotPipeline: -Take screenshot of item ------------------------ +Take a screenshot of an item +---------------------------- This example demonstrates how to use :doc:`coroutine syntax ` in the :meth:`process_item` method. -This item pipeline makes a request to a locally-running instance of Splash_ to +This item pipeline makes a request to a locally running instance of Splash_ to render a screenshot of the item URL. After the request response is downloaded, the item pipeline saves the screenshot to a file and adds the filename to the item. @@ -195,8 +195,8 @@ item. class ScreenshotPipeline: - """Pipeline that uses Splash to render screenshot of - every Scrapy item.""" + """Pipeline that uses Splash to render a screenshot of every Scrapy + item.""" SPLASH_URL = "http://localhost:8050/render.png?url={}" @@ -215,16 +215,17 @@ item. response = await self.crawler.engine.download_async(request) if response.status != 200: - # Error happened, return item. + # An error occurred, so return the item. return item - # Save screenshot to file, filename will be hash of url. + # Save the screenshot to a file; the filename is the hash of the + # URL. url = adapter["url"] url_hash = hashlib.md5(url.encode("utf8")).hexdigest() filename = f"{url_hash}.png" Path(filename).write_bytes(response.body) - # Store filename in item. + # Store the filename in the item. adapter["screenshot_filename"] = filename return item @@ -233,9 +234,9 @@ item. Duplicates filter ----------------- -A filter that looks for duplicate items, and drops those items that were -already processed. Let's say that our items have a unique id, but our spider -returns multiples items with the same id: +This filter looks for duplicate items and drops those that were already +processed. Let's say that our items have a unique id, but our spider returns +multiple items with the same id: .. code-block:: python diff --git a/docs/topics/items.rst b/docs/topics/items.rst index 0892dd839..df6030eed 100644 --- a/docs/topics/items.rst +++ b/docs/topics/items.rst @@ -8,7 +8,7 @@ Items :synopsis: Item and Field classes The main goal in scraping is to extract structured data from unstructured -sources, typically, web pages. :ref:`Spiders ` may return the +sources, typically web pages. :ref:`Spiders ` may return the extracted data as `items`, Python objects that define key-value pairs. Scrapy supports :ref:`multiple types of items `. When you create an @@ -21,10 +21,10 @@ receives an item, your code should :ref:`work for any item type Item Types ========== -Scrapy supports the following types of items, via the `itemadapter`_ library: +Scrapy supports the following types of items via the `itemadapter`_ library: :ref:`dictionaries `, :ref:`Item objects `, -:ref:`dataclass objects `, :ref:`attrs objects ` -and :ref:`Pydantic models `. +:ref:`dataclass objects `, :ref:`attrs objects +`, and :ref:`Pydantic models `. .. _itemadapter: https://github.com/scrapy/itemadapter @@ -50,16 +50,16 @@ make it the most feature-complete item type: :class:`Item` objects replicate the standard :class:`dict` API, including its ``__init__`` method. -:class:`Item` allows the defining of field names, so that: +:class:`Item` allows you to define field names, so that: -- :class:`KeyError` is raised when using undefined field names (i.e. - prevents typos going unnoticed) +- :class:`KeyError` is raised when using undefined field names (i.e. prevents + typos from going unnoticed) - :ref:`Item exporters ` can export all fields by default even if the first scraped object does not have values for all of them -:class:`Item` also allows the defining of field metadata, which can be used to +:class:`Item` also lets you define field metadata, which can be used to :ref:`customize serialization `. :mod:`scrapy.utils.trackref` tracks :class:`Item` objects to help find memory @@ -81,11 +81,12 @@ Example: Dataclass objects ----------------- -:func:`~dataclasses.dataclass` allows the defining of item classes with field names, -so that :ref:`item exporters ` can export all fields by -default even if the first scraped object does not have values for all of them. +:func:`~dataclasses.dataclass` allows you to define item classes with field +names, so that :ref:`item exporters ` can export all fields +by default even if the first scraped object does not have values for all of +them. -Additionally, ``dataclass`` items also allow you to: +Additionally, ``dataclass`` items allow you to: * define the type and default value of each defined field. @@ -111,18 +112,18 @@ Example: attr.s objects -------------- -:func:`attr.s` allows the defining of item classes with field names, -so that :ref:`item exporters ` can export all fields by -default even if the first scraped object does not have values for all of them. +:func:`attr.s` allows you to define item classes with field names, so that +:ref:`item exporters ` can export all fields by default even +if the first scraped object does not have values for all of them. -Additionally, ``attr.s`` items also allow to: +Additionally, ``attr.s`` items allow you to: * define the type and default value of each defined field. * define custom field :ref:`metadata `, which can be used to :ref:`customize serialization `. -In order to use this type, the :doc:`attrs package ` needs to be installed. +To use this type, the :doc:`attrs package ` needs to be installed. Example: @@ -199,9 +200,9 @@ Item subclasses are declared using a simple class definition syntax and tags = scrapy.Field() last_updated = scrapy.Field(serializer=str) -.. note:: Those familiar with `Django`_ will notice that Scrapy Items are - declared similar to `Django Models`_, except that Scrapy Items are much - simpler as there is no concept of different field types. +.. note:: Those familiar with `Django`_ will notice that Scrapy items are + declared similarly to `Django Models`_, except that Scrapy items are much + simpler as there is no concept of different field types. .. _Django: https://www.djangoproject.com/ .. _Django Models: https://docs.djangoproject.com/en/dev/topics/db/models/ @@ -213,19 +214,19 @@ Declaring fields ---------------- :class:`Field` objects are used to specify metadata for each field. For -example, the serializer function for the ``last_updated`` field illustrated in -the example above. +example, they can store the serializer function for the ``last_updated`` field +illustrated above. You can specify any kind of metadata for each field. There is no restriction on -the values accepted by :class:`Field` objects. For this same -reason, there is no reference list of all available metadata keys. Each key -defined in :class:`Field` objects could be used by a different component, and -only those components know about it. You can also define and use any other -:class:`Field` key in your project too, for your own needs. The main goal of -:class:`Field` objects is to provide a way to define all field metadata in one -place. Typically, those components whose behaviour depends on each field use -certain field keys to configure that behaviour. You must refer to their -documentation to see which metadata keys are used by each component. +the values accepted by :class:`Field` objects. For this same reason, there is +no reference list of all available metadata keys. Each key defined in +:class:`Field` objects could be used by a different component, and only those +components know about it. You can also define and use any other :class:`Field` +key in your project too, for your own needs. The main goal of :class:`Field` +objects is to provide a way to define all field metadata in one place. +Typically, those components whose behavior depends on each field use certain +field keys to configure that behavior. You must refer to their documentation to +see which metadata keys are used by each component. It's important to note that the :class:`Field` objects used to declare the item do not stay assigned as class attributes. Instead, they can be accessed through @@ -253,7 +254,7 @@ Working with Item objects .. skip: start Here are some examples of common tasks performed with items, using the -``Product`` item :ref:`declared above `. You will +``Product`` item :ref:`declared above `. You will notice the API is very similar to the :class:`dict` API. Creating items @@ -393,8 +394,8 @@ Creating items from dicts: Extending Item subclasses ------------------------- -You can extend Items (to add more fields or to change some metadata for some -fields) by declaring a subclass of your original Item. +You can extend items (to add more fields or to change some metadata for some +fields) by declaring a subclass of your original item. For example: @@ -425,7 +426,7 @@ Supporting All Item Types In code that receives an item, such as methods of :ref:`item pipelines ` or :ref:`spider middlewares -`, it is a good practice to use the +`, it is good practice to use the :class:`~itemadapter.ItemAdapter` class to write code that works for any supported item type. diff --git a/docs/topics/jobs.rst b/docs/topics/jobs.rst index c3043204b..b10546c33 100644 --- a/docs/topics/jobs.rst +++ b/docs/topics/jobs.rst @@ -7,7 +7,7 @@ Jobs: pausing and resuming crawls Sometimes, for big sites, it's desirable to pause crawls and be able to resume them later. -Scrapy supports this functionality out of the box by providing the following +Scrapy supports this functionality out of the box through the following facilities: * a scheduler that persists scheduled requests on disk @@ -44,7 +44,7 @@ To start a spider with persistence support enabled, run it like this:: scrapy crawl somespider -s JOBDIR=crawls/somespider-1 -Then, you can stop the spider safely at any time (by pressing Ctrl-C or sending +Then you can stop the spider safely at any time (by pressing Ctrl-C or sending a signal), and resume it later by issuing the same command:: scrapy crawl somespider -s JOBDIR=crawls/somespider-1 @@ -54,11 +54,11 @@ a signal), and resume it later by issuing the same command:: Keeping persistent state between batches ======================================== -Sometimes you'll want to keep some persistent spider state between pause/resume -batches. You can use the ``spider.state`` attribute for that, which should be a -dict. There's :ref:`a built-in extension ` -that takes care of serializing, storing and loading that attribute from the job -directory, when the spider starts and stops. +Sometimes you'll want to keep persistent spider state between pause/resume +batches. Use the ``spider.state`` attribute for that. It should be a dict. +There's :ref:`a built-in extension ` that +takes care of serializing, storing and loading that attribute from the job +directory when the spider starts and stops. Here's an example of a callback that uses the spider state (other spider code is omitted for brevity): @@ -72,8 +72,8 @@ is omitted for brevity): Persistence gotchas =================== -There are a few things to keep in mind if you want to be able to use the Scrapy -persistence support: +There are a few things to keep in mind if you want to use Scrapy's persistence +support: Pause limitations ----------------- @@ -91,11 +91,11 @@ version that wrote them. A job must be resumed with the same Scrapy version that paused it; after upgrading or downgrading Scrapy, start a new job with a new job directory. -Cookies expiration ------------------- +Cookie expiration +----------------- -Cookies may expire. So, if you don't resume your spider quickly the requests -scheduled may no longer work. This won't be an issue if your spider doesn't rely +Cookies may expire, so if you don't resume your spider quickly, the scheduled +requests may no longer work. This won't be an issue if your spider doesn't rely on cookies. .. _request-serialization: diff --git a/docs/topics/leaks.rst b/docs/topics/leaks.rst index 0a8b146d2..e77f73901 100644 --- a/docs/topics/leaks.rst +++ b/docs/topics/leaks.rst @@ -5,55 +5,53 @@ Debugging memory leaks ====================== In Scrapy, objects such as requests, responses and items have a finite -lifetime: they are created, used for a while, and finally destroyed. +lifetime: they are created, used for a while and finally destroyed. -From all those objects, the Request is probably the one with the longest -lifetime, as it stays waiting in the Scheduler queue until it's time to process -it. For more info see :ref:`topics-architecture`. +Of all those objects, the Request usually has the longest lifetime because it +waits in the Scheduler queue until it's time to process it. For more +information, see :ref:`topics-architecture`. -As these Scrapy objects have a (rather long) lifetime, there is always the risk -of accumulating them in memory without releasing them properly and thus causing -what is known as a "memory leak". +Because these Scrapy objects have a comparatively long lifetime, there is +always the risk of accumulating them in memory without releasing them properly +and thus causing what is known as a "memory leak". -To help debugging memory leaks, Scrapy provides a built-in mechanism for -tracking objects references called :ref:`trackref `, -and you can also use a third-party library called :ref:`muppy -` for more advanced memory debugging (see below for more -info). Both mechanisms must be used from the :ref:`Telnet Console -`. +To help debug memory leaks, Scrapy provides a built-in mechanism for tracking +object references called :ref:`trackref `, and you can +also use a third-party library called :ref:`muppy ` for +more advanced memory debugging (see below for more information). Both +mechanisms must be used from the :ref:`Telnet Console `. Common causes of memory leaks ============================= It happens quite often (sometimes by accident, sometimes on purpose) that the -Scrapy developer passes objects referenced in Requests (for example, using the -:attr:`~scrapy.Request.cb_kwargs` or :attr:`~scrapy.Request.meta` -attributes or the request callback function) and that effectively bounds the -lifetime of those referenced objects to the lifetime of the Request. This is, -by far, the most common cause of memory leaks in Scrapy projects, and a quite -difficult one to debug for newcomers. +Scrapy developer passes objects referenced in requests—for example, through the +:attr:`~scrapy.Request.cb_kwargs` or :attr:`~scrapy.Request.meta` attributes or +the request callback function—and that effectively ties the lifetime of those +referenced objects to the lifetime of the request. This is, by far, the most +common cause of memory leaks in Scrapy projects, and it can be quite difficult +for newcomers to debug. -In big projects, the spiders are typically written by different people and some -of those spiders could be "leaking" and thus affecting the rest of the other -(well-written) spiders when they get to run concurrently, which, in turn, -affects the whole crawling process. +In big projects, spiders are typically written by different people, and some of +those spiders could be "leaking" and thus affecting the rest of the +(well-written) spiders when they run concurrently, which, in turn, affects the +whole crawling process. -The leak could also come from a custom middleware, pipeline or extension that -you have written, if you are not releasing the (previously allocated) resources -properly. For example, allocating resources on :signal:`spider_opened` -but not releasing them on :signal:`spider_closed` may cause problems if -you're running :ref:`multiple spiders per process `. +The leak could also come from a custom middleware, pipeline, or extension that +you wrote if you are not releasing the previously allocated resources properly. +For example, allocating resources on :signal:`spider_opened` but not releasing +them on :signal:`spider_closed` may cause problems if you're running +:ref:`multiple spiders per process `. Too Many Requests? ------------------ By default Scrapy keeps the request queue in memory; it includes -:class:`~scrapy.Request` objects and all objects -referenced in Request attributes (e.g. in :attr:`~scrapy.Request.cb_kwargs` -and :attr:`~scrapy.Request.meta`). -While not necessarily a leak, this can take a lot of memory. Enabling -:ref:`persistent job queue ` could help keeping memory usage -in control. +:class:`~scrapy.Request` objects and all objects referenced in request +attributes (for example, :attr:`~scrapy.Request.cb_kwargs` and +:attr:`~scrapy.Request.meta`). While this is not necessarily a leak, it can +consume a lot of memory. Enabling the :ref:`persistent job queue ` +can help keep memory usage under control. .. _topics-leaks-trackrefs: @@ -63,8 +61,8 @@ Debugging memory leaks with ``trackref`` .. skip: start :mod:`scrapy.utils.trackref` is a module provided by Scrapy to debug the most -common cases of memory leaks. It basically tracks the references to all live -Request, Response, Item, Spider and Selector objects. +common cases of memory leaks. It tracks the references to all live Request, +Response, Item, Spider and Selector objects. You can enter the telnet console and inspect how many objects (of the classes mentioned above) are currently alive using the ``prefs()`` function which is an @@ -85,16 +83,15 @@ alias to the :func:`~scrapy.utils.trackref.print_live_refs` function: Request 878 oldest: 7s ago As you can see, that report also shows the "age" of the oldest object in each -class. If you're running multiple spiders per process chances are you can -figure out which spider is leaking by looking at the oldest request or response. -You can get the oldest object of each class using the +class. If you're running multiple spiders per process, chances are you can +figure out which spider is leaking by looking at the oldest request or +response. You can get the oldest object of each class using the :func:`~scrapy.utils.trackref.get_oldest` function (from the telnet console). Which objects are tracked? -------------------------- -The objects tracked by ``trackref`` are all from these classes (and all its -subclasses): +``trackref`` tracks objects from these classes (and all their subclasses): * :class:`scrapy.Request` * :class:`scrapy.http.Response` @@ -116,9 +113,9 @@ Suppose we have some spider with a line similar to this one: cb_kwargs={"referer": response}, ) -That line is passing a response reference inside a request which effectively -ties the response lifetime to the requests' one, and that would definitely -cause memory leaks. +That line passes a response reference inside a request, which effectively ties +the response lifetime to the request's lifetime, and that will definitely cause +memory leaks. Let's see how we can discover the cause (without knowing it a priori, of course) by using the ``trackref`` tool. @@ -139,10 +136,10 @@ references: The fact that there are so many live responses (and that they're so old) is definitely suspicious, as responses should have a relatively short lifetime -compared to Requests. The number of responses is similar to the number -of requests, so it looks like they are tied in a some way. We can now go -and check the code of the spider to discover the nasty line that is -generating the leaks (passing response references inside requests). +compared to requests. The number of responses is similar to the number of +requests, so it looks like they are tied in some way. We can now check the code +of the spider to discover the line that is generating the leaks (passing +response references inside requests). Sometimes extra information about live objects can be helpful. Let's check the oldest response: @@ -154,8 +151,8 @@ Let's check the oldest response: >>> r.url 'http://www.somenastyspider.com/product.php?pid=123' -If you want to iterate over all objects, instead of getting the oldest one, you -can use the :func:`scrapy.utils.trackref.iter_all` function: +If you want to iterate over all objects instead of getting only the oldest one, +you can use the :func:`scrapy.utils.trackref.iter_all` function: .. code-block:: pycon @@ -168,11 +165,10 @@ can use the :func:`scrapy.utils.trackref.iter_all` function: Too many spiders? ----------------- -If your project has too many spiders executed in parallel, -the output of ``prefs()`` can be difficult to read. -For this reason, that function has a ``ignore`` argument which can be used to -ignore a particular class (and all its subclasses). For -example, this won't show any live references to spiders: +If your project has too many spiders executed in parallel, the output of +``prefs()`` can be difficult to read. For this reason, that function has an +``ignore`` argument that you can use to omit a particular class (and all its +subclasses). For example, this won't show any live references to spiders: .. code-block:: pycon @@ -202,14 +198,13 @@ Here are the functions available in the :mod:`~scrapy.utils.trackref` module. Debugging memory leaks with muppy ================================= -``trackref`` provides a very convenient mechanism for tracking down memory -leaks, but it only keeps track of the objects that are more likely to cause -memory leaks. However, there are other cases where the memory leaks could come -from other (more or less obscure) objects. If this is your case, and you can't -find your leaks using ``trackref``, you still have another resource: the muppy -library. +``trackref`` provides a convenient mechanism for tracking down memory leaks, +but it only keeps track of the objects that are more likely to cause them. +However, sometimes leaks come from other (more or less obscure) objects. If +that happens and you can't find your leaks using ``trackref``, you still have +another resource: the muppy library. -You can use muppy from `Pympler`_. +muppy is available as part of `Pympler`_. .. _Pympler: https://pypi.org/project/Pympler/ @@ -217,8 +212,8 @@ If you use ``pip``, you can install muppy with the following command:: pip install Pympler -Here's an example to view all Python objects available in -the heap using muppy: +Here's an example that shows all Python objects available in the heap using +muppy: .. skip: start .. code-block:: pycon @@ -250,7 +245,7 @@ the heap using muppy: .. skip: end -For more info about muppy, refer to the `muppy documentation`_. +For more information about muppy, refer to the `muppy documentation`_. .. _muppy documentation: https://pythonhosted.org/Pympler/muppy.html @@ -259,10 +254,10 @@ For more info about muppy, refer to the `muppy documentation`_. Leaks without leaks =================== -Sometimes, you may notice that the memory usage of your Scrapy process will -only increase, but never decrease. Unfortunately, this could happen even -though neither Scrapy nor your project are leaking memory. This is due to a -(not so well) known problem of Python, which may not return released memory to +Sometimes you may notice that the memory usage of your Scrapy process only +increases and never decreases. Unfortunately, this could happen even though +neither Scrapy nor your project are leaking memory. This is due to a +not-so-well-known problem in Python, which may not return released memory to the operating system in some cases. For more information on this issue see: * `Python Memory Management `_ @@ -283,6 +278,6 @@ completely. To quote the paper: .. _this paper: https://www.evanjones.ca/memoryallocator/ -To keep memory consumption reasonable you can split the job into several -smaller jobs or enable :ref:`persistent job queue ` -and stop/start spider from time to time. +To keep memory consumption reasonable, you can split the job into several +smaller jobs or enable the :ref:`persistent job queue ` and stop +and start the spider from time to time. diff --git a/docs/topics/link-extractors.rst b/docs/topics/link-extractors.rst index c1d872dd0..1985c0695 100644 --- a/docs/topics/link-extractors.rst +++ b/docs/topics/link-extractors.rst @@ -16,9 +16,10 @@ list of matching :class:`~scrapy.link.Link` objects from a Link extractors are used in :class:`~scrapy.spiders.CrawlSpider` spiders through a set of :class:`~scrapy.spiders.Rule` objects. -You can also use link extractors in regular spiders. For example, you can instantiate -:class:`LinkExtractor ` into a class -variable in your spider, and use it from your spider callbacks: +You can also use link extractors in regular spiders. For example, you can +instantiate :class:`LinkExtractor +` as a class variable in your +spider and use it from your spider callbacks: .. code-block:: python diff --git a/docs/topics/loaders.rst b/docs/topics/loaders.rst index 314e2c1ae..3296b5e48 100644 --- a/docs/topics/loaders.rst +++ b/docs/topics/loaders.rst @@ -8,17 +8,18 @@ Item Loaders :synopsis: Item Loader class Item Loaders provide a convenient mechanism for populating scraped :ref:`items -`. Even though items can be populated directly, Item Loaders provide a -much more convenient API for populating them from a scraping process, by automating -some common tasks like parsing the raw extracted data before assigning it. +`. Even though items can be populated directly, Item Loaders +provide a more convenient API for populating them from a scraping process by +automating common tasks like parsing the raw extracted data before assigning +it. In other words, :ref:`items ` provide the *container* of scraped data, while Item Loaders provide the mechanism for *populating* that container. Item Loaders are designed to provide a flexible, efficient and easy mechanism -for extending and overriding different field parsing rules, either by spider, -or by source format (HTML, XML, etc) without becoming a nightmare to maintain. +for extending and overriding different field parsing rules, either by spider or +by source format (HTML, XML, etc.) without becoming a nightmare to maintain. .. note:: Item Loaders are an extension of the itemloaders_ library that make it easier to work with Scrapy by adding support for @@ -80,9 +81,9 @@ and finally the ``last_updated`` field is populated directly with a literal valu (``today``) using a different method: :meth:`~ItemLoader.add_value`. Finally, when all data is collected, the :meth:`ItemLoader.load_item` method is -called which actually returns the item populated with the data -previously extracted and collected with the :meth:`~ItemLoader.add_xpath`, -:meth:`~ItemLoader.add_css`, and :meth:`~ItemLoader.add_value` calls. +called, which returns the item populated with the data previously extracted and +collected with the :meth:`~ItemLoader.add_xpath`, :meth:`~ItemLoader.add_css`, +and :meth:`~ItemLoader.add_value` calls. .. _topics-loaders-dataclass: @@ -117,15 +118,15 @@ Input and Output processors =========================== An Item Loader contains one input processor and one output processor for each -(item) field. The input processor processes the extracted data as soon as it's -received (through the :meth:`~ItemLoader.add_xpath`, :meth:`~ItemLoader.add_css` or -:meth:`~ItemLoader.add_value` methods) and the result of the input processor is -collected and kept inside the ItemLoader. After collecting all data, the -:meth:`ItemLoader.load_item` method is called to populate and get the populated -:ref:`item object `. That's when the output processor is -called with the data previously collected (and processed using the input -processor). The result of the output processor is the final value that gets -assigned to the item. +field. The input processor processes the extracted data as soon as it's +received (through the :meth:`~ItemLoader.add_xpath`, +:meth:`~ItemLoader.add_css`, or :meth:`~ItemLoader.add_value` methods), and the +result of the input processor is collected and kept inside the Item Loader. +After collecting all data, :meth:`ItemLoader.load_item` is called to populate +and return the populated :ref:`item object `. That's when the +output processor is called with the data previously collected and processed by +the input processor. The result of the output processor is the final value that +gets assigned to the item. Let's see an example to illustrate how the input and output processors are called for a particular field (the same applies for any other field): @@ -151,33 +152,32 @@ So what happens is: data collected in (1) (if any). 3. This case is similar to the previous ones, except that the data is extracted - from the ``css`` CSS selector, and passed through the same *input - processor* used in (1) and (2). The result of the input processor is appended to the + from the ``css`` CSS selector and passed through the same *input processor* + used in (1) and (2). The result of the input processor is appended to the data collected in (1) and (2) (if any). 4. This case is also similar to the previous ones, except that the value to be collected is assigned directly, instead of being extracted from a XPath - expression or a CSS selector. - However, the value is still passed through the input processors. In this - case, since the value is not iterable it is converted to an iterable of a - single element before passing it to the input processor, because input - processor always receive iterables. + expression or a CSS selector. However, the value is still passed through the + input processors. In this case, since the value is not iterable it is + converted to an iterable of a single element before passing it to the input + processor because input processors always receive iterables. 5. The data collected in steps (1), (2), (3) and (4) is passed through the *output processor* of the ``name`` field. The result of the output processor is the value assigned to the ``name`` field in the item. -It's worth noticing that processors are just callable objects, which are called -with the data to be parsed, and return a parsed value. So you can use any +It's worth noting that processors are just callable objects that are called +with the data to be parsed and return a parsed value. So you can use any function as input or output processor. The only requirement is that they must accept one (and only one) positional argument, which will be an iterable. .. note:: Both input and output processors must receive an iterable as their first argument. The output of those functions can be anything. The result of - input processors will be appended to an internal list (in the Loader) - containing the collected values (for that field). The result of the output - processors is the value that will be finally assigned to the item. + input processors will be appended to an internal list (in the loader) + containing the collected values for that field. The result of the output + processors is the value that will be assigned to the item. The other thing you need to keep in mind is that the values returned by input processors are collected internally (in lists) and then passed to output diff --git a/docs/topics/logging.rst b/docs/topics/logging.rst index b486eab63..9cb81d48b 100644 --- a/docs/topics/logging.rst +++ b/docs/topics/logging.rst @@ -4,9 +4,9 @@ Logging ======= -Scrapy uses :mod:`logging` for event logging. We'll -provide some simple examples to get you started, but for more advanced -use-cases it's strongly suggested to read thoroughly its documentation. +Scrapy uses :mod:`logging` for event logging. We'll provide some simple +examples to get you started, but for more advanced use-cases it's strongly +suggested to read its documentation thoroughly. Logging works out of the box, and can be configured to some extent with the Scrapy settings listed in :ref:`topics-logging-settings`. @@ -21,8 +21,8 @@ Scrapy from scripts as described in :ref:`run-from-script`. Log levels ========== -Python's builtin logging defines 5 different levels to indicate the severity of a -given log message. Here are the standard ones, listed in decreasing order: +Python's built-in logging defines 5 different levels to indicate the severity +of a given log message. Here are the standard ones, listed in decreasing order: 1. ``logging.CRITICAL`` - for critical errors (highest severity) 2. ``logging.ERROR`` - for regular errors @@ -57,10 +57,10 @@ example, a common practice is to create different loggers for every module). These loggers can be configured independently, and they allow hierarchical constructions. -The previous examples use the root logger behind the scenes, which is a top level -logger where all messages are propagated to (unless otherwise specified). Using -``logging`` helpers is merely a shortcut for getting the root logger -explicitly, so this is also an equivalent of the last snippets: +The previous examples use the root logger behind the scenes, which is a +top-level logger where all messages are propagated to (unless otherwise +specified). Using ``logging`` helpers is merely a shortcut for getting the root +logger explicitly, so this is also an equivalent of the last snippets: .. code-block:: python @@ -80,8 +80,8 @@ You can use a different logger just by getting its name with the logger.warning("This is a warning") Finally, you can ensure having a custom logger for any module you're working on -by using the ``__name__`` variable, which is populated with current module's -path: +by using the ``__name__`` variable, which is populated with the current +module's path: .. code-block:: python @@ -321,9 +321,8 @@ filter out unwanted messages: if match: return False -A project-level filter may be attached to the root -handler created by Scrapy, this is a wieldy way to -filter all loggers in different parts of the project +A project-level filter may be attached to the root handler created by Scrapy. +This is a useful way to filter all loggers in different parts of the project (middlewares, spider, etc.): .. code-block:: python diff --git a/docs/topics/media-pipeline.rst b/docs/topics/media-pipeline.rst index 576feae7e..7441f5303 100644 --- a/docs/topics/media-pipeline.rst +++ b/docs/topics/media-pipeline.rst @@ -14,15 +14,15 @@ typically you'll either use the Files Pipeline or the Images Pipeline. Both pipelines implement these features: -* Avoid re-downloading media that was downloaded recently -* Specifying where to store the media (filesystem directory, FTP server, Amazon S3 bucket, - Google Cloud Storage bucket) +* Avoid re-downloading media that have been downloaded recently +* Specify where to store the media (filesystem directory, FTP server, Amazon S3 + bucket, Google Cloud Storage bucket) The Images Pipeline has a few extra functions for processing images: -* Convert all downloaded images to a common format (JPG) and mode (RGB) +* Convert all downloaded images to a common format (JPEG) and mode (RGB) * Thumbnail generation -* Check images width/height to make sure they meet a minimum constraint +* Check images' width/height to make sure they meet a minimum constraint The pipelines also keep an internal queue of those media URLs which are currently being scheduled for download, and connect those responses that arrive containing @@ -49,9 +49,9 @@ this: 4. When the files are downloaded, another field (``files``) will be populated with the results. This field will contain a list of dicts with information about the downloaded files, such as the downloaded path, the original - scraped url (taken from the ``file_urls`` field), the file checksum and the file status. - The files in the list of the ``files`` field will retain the same order of - the original ``file_urls`` field. If some file failed downloading, an + scraped URL (taken from the ``file_urls`` field), the file checksum and the + file status. The files in the ``files`` field will retain the same order as + in the original ``file_urls`` field. If some file failed downloading, an error will be logged and the file won't be present in the ``files`` field. @@ -285,8 +285,8 @@ policy: For more information, see `canned ACLs`_ in the Amazon S3 Developer Guide. You can also use other S3-like storages. Storages like self-hosted `Minio`_ or -`Zenko CloudServer`_. All you need to do is set endpoint option in you Scrapy -settings: +`Zenko CloudServer`_. All you need to do is set the endpoint option in your +Scrapy settings: .. code-block:: python @@ -434,13 +434,14 @@ specifies the delay in number of days: The default value for both settings is 90 days. -If you have pipeline that subclasses FilesPipeline and you'd like to have -different setting for it you can set setting keys preceded by uppercase -class name. E.g. given pipeline class called MyPipeline you can set setting key: +If you have a pipeline that subclasses FilesPipeline and you'd like to have a +different setting for it, you can set setting keys preceded by the uppercase +class name. For example, given a pipeline class called MyPipeline you can set +the setting key: MYPIPELINE_FILES_EXPIRES = 180 -and pipeline class MyPipeline will have expiration time set to 180. +and the pipeline class MyPipeline will have an expiration time set to 180. The last modified time from the file is used to determine the age of the file in days, which is then compared to the set expiration time to determine if the file is expired. diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rst index 1d97e39b6..0fae3643a 100644 --- a/docs/topics/request-response.rst +++ b/docs/topics/request-response.rst @@ -7,8 +7,8 @@ Requests and Responses .. module:: scrapy.http :synopsis: Request and Response classes -Scrapy uses :class:`~scrapy.Request` and :class:`Response` objects for crawling web -sites. +Scrapy uses :class:`~scrapy.Request` and :class:`Response` objects for crawling +websites. Typically, :class:`~scrapy.Request` objects are generated in the spiders and pass across the system until they reach the Downloader, which executes the request @@ -713,7 +713,7 @@ Errbacks ======== The errback of a request is a function that will be called when an exception -is raise while processing it. +is raised while processing it. It receives a :exc:`~twisted.python.failure.Failure` as first parameter and can be used to track connection establishment timeouts, DNS errors etc. diff --git a/docs/topics/scheduler.rst b/docs/topics/scheduler.rst index b79d6de1f..91e13a0ba 100644 --- a/docs/topics/scheduler.rst +++ b/docs/topics/scheduler.rst @@ -6,10 +6,10 @@ Scheduler .. module:: scrapy.core.scheduler -The scheduler component receives requests from the :ref:`engine ` -and stores them into persistent and/or non-persistent data structures. -It also gets those requests and feeds them back to the engine when it -asks for a next request to be downloaded. +The scheduler component receives requests from the :ref:`engine +` and stores them into persistent and/or non-persistent data +structures. It also gets those requests and feeds them back to the engine when +it asks for the next request to be downloaded. Overriding the default scheduler diff --git a/docs/topics/selectors.rst b/docs/topics/selectors.rst index 6d4954c2d..bec03578c 100644 --- a/docs/topics/selectors.rst +++ b/docs/topics/selectors.rst @@ -26,13 +26,14 @@ used with HTML. `CSS`_ is a language for applying styles to HTML documents. It defines selectors to associate those styles with specific HTML elements. .. note:: - Scrapy Selectors is a thin wrapper around `parsel`_ library; the purpose of - this wrapper is to provide better integration with Scrapy Response objects. + Scrapy selectors are a thin wrapper around the `parsel`_ library; the + purpose of this wrapper is to provide better integration with Scrapy + Response objects. `parsel`_ is a stand-alone web scraping library which can be used without - Scrapy. It uses `lxml`_ library under the hood, and implements an - easy API on top of lxml API. It means Scrapy selectors are very similar - in speed and parsing accuracy to lxml. + Scrapy. It uses the `lxml`_ library under the hood and implements an easy + API on top of the lxml API. This means Scrapy selectors have similar speed + and parsing accuracy to lxml. .. _BeautifulSoup: https://www.crummy.com/software/BeautifulSoup/ .. _lxml: https://lxml.de/ diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst index e58815fad..6093c0986 100644 --- a/docs/topics/settings.rst +++ b/docs/topics/settings.rst @@ -4,7 +4,7 @@ Settings ======== -The Scrapy settings allows you to customize the behaviour of all Scrapy +The Scrapy settings allow you to customize the behaviour of all Scrapy components, including the core, extensions, pipelines and spiders themselves. The infrastructure of the settings provides a global namespace of key-value mappings @@ -1192,7 +1192,7 @@ response was not properly finished. If ``True``, these responses raise a responses are passed through and the flag ``dataloss`` is added to the response, i.e.: ``'dataloss' in response.flags`` is ``True``. -Optionally, this can be set per-request basis by using the +Optionally, this can be set on a per-request basis by using the :reqmeta:`download_fail_on_dataloss` Request.meta key to ``False``. .. note:: diff --git a/docs/topics/shell.rst b/docs/topics/shell.rst index 42c5bd169..71fb1ca37 100644 --- a/docs/topics/shell.rst +++ b/docs/topics/shell.rst @@ -14,7 +14,7 @@ and what data they extract from the web pages you're trying to scrape. It allows you to interactively test your expressions while you're writing your spider, without having to run the spider to test every change. -Once you get familiarized with the Scrapy shell, you'll see that it's an +Once you become familiar with the Scrapy shell, you'll see that it's an invaluable tool for developing and debugging your spiders. .. _shell-config: @@ -33,11 +33,10 @@ output, and more. Scrapy also has support for `bpython`_ via the :ref:`bpython ` extra, and will try to use it where neither ptpython nor IPython is available. -Through Scrapy's settings you can configure it to use any one of -``ptpython``, ``ipython``, ``bpython`` or the standard ``python`` shell, -regardless of which are installed. This is done by setting the -``SCRAPY_PYTHON_SHELL`` environment variable; or by defining it in your -:ref:`scrapy.cfg `: +Through Scrapy's settings you can configure it to use any one of ``ptpython``, +``ipython``, ``bpython`` or the standard ``python`` shell, regardless of which +are installed. This is done by setting the ``SCRAPY_PYTHON_SHELL`` environment +variable or by defining it in your :ref:`scrapy.cfg `: .. code-block:: ini diff --git a/docs/topics/signals.rst b/docs/topics/signals.rst index 0a85c3c05..df55c1f09 100644 --- a/docs/topics/signals.rst +++ b/docs/topics/signals.rst @@ -154,7 +154,7 @@ scheduler_empty Sent whenever the engine asks for a pending request from the :ref:`scheduler ` (i.e. calls its :meth:`~scrapy.core.scheduler.BaseScheduler.next_request` method) and the - scheduler returns none. + scheduler returns None. See :ref:`start-requests-lazy` for an example. @@ -174,12 +174,11 @@ Item signals ------------ .. note:: - As at max :setting:`CONCURRENT_ITEMS` items are processed in - parallel, many deferreds are fired together using - :class:`~twisted.internet.defer.DeferredList`. Hence the next - batch waits for the :class:`~twisted.internet.defer.DeferredList` - to fire and then runs the respective item signal handler for - the next batch of scraped items. + At most :setting:`CONCURRENT_ITEMS` items are processed in parallel, many + deferreds are fired together using + :class:`~twisted.internet.defer.DeferredList`. Hence the next batch waits + for the :class:`~twisted.internet.defer.DeferredList` to fire and then runs + the respective item signal handler for the next batch of scraped items. item_scraped ~~~~~~~~~~~~ @@ -271,12 +270,12 @@ spider_closed :param spider: the spider which has been closed :type spider: :class:`~scrapy.Spider` object - :param reason: a string which describes the reason why the spider was closed. If - it was closed because the spider has completed scraping, the reason - is ``'finished'``. Otherwise, if the spider was manually closed by - calling the ``close_spider`` engine method, then the reason is the one - passed in the ``reason`` argument of that method (which defaults to - ``'cancelled'``). If the engine was shutdown (for example, by hitting + :param reason: a string which describes the reason why the spider was + closed. If it was closed because the spider has completed scraping, the + reason is ``'finished'``. Otherwise, if the spider was manually closed + by calling the ``close_spider`` engine method, then the reason is the + one passed in the ``reason`` argument of that method (which defaults to + ``'cancelled'``). If the engine was shut down (for example, by hitting Ctrl-C to stop it) the reason will be ``'shutdown'``. :type reason: str @@ -453,11 +452,11 @@ request_reached_downloader .. signal:: request_reached_downloader .. function:: request_reached_downloader(request, spider) - Sent when a :class:`~scrapy.Request` reached downloader. + Sent when a :class:`~scrapy.Request` reached the downloader. This signal does not support :ref:`asynchronous handlers `. - :param request: the request that reached downloader + :param request: the request that reached the downloader :type request: :class:`~scrapy.Request` object :param spider: the spider that yielded the request @@ -486,11 +485,11 @@ bytes_received .. signal:: bytes_received .. function:: bytes_received(data, request, spider) - Sent by some download handlers when a group of bytes is - received for a specific request. This signal might be fired multiple - times for the same request, with partial data each time. For instance, - a possible scenario for a 25 kb response would be two signals fired - with 10 kb of data, and a final one with 5 kb of data. + Sent by some download handlers when a group of bytes is received for a + specific request. This signal might be fired multiple times for the same + request, with partial data each time. For instance, a possible scenario for + a 25 KB response would be two signals fired with 10 KB of data, and a final + one with 5 KB of data. Handlers for this signal can stop the download of a response while it is in progress by raising the :exc:`~scrapy.exceptions.StopDownload` diff --git a/docs/topics/spider-middleware.rst b/docs/topics/spider-middleware.rst index 1c9ee0c77..8c244d0f7 100644 --- a/docs/topics/spider-middleware.rst +++ b/docs/topics/spider-middleware.rst @@ -4,10 +4,10 @@ Spider Middleware ================= -The spider middleware is a framework of hooks into Scrapy's spider processing -mechanism where you can plug custom functionality to process the responses that -are sent to :ref:`topics-spiders` for processing and to process the requests -and items that are generated from spiders. +Spider middleware is a framework of hooks into Scrapy's spider processing +mechanism where you can plug in custom functionality to process the responses +that are sent to :ref:`topics-spiders` for processing, and to process the +requests and items that are generated by spiders. .. _topics-spider-middleware-setting: @@ -16,7 +16,7 @@ Activating a spider middleware To activate a spider middleware component, add it to the :setting:`SPIDER_MIDDLEWARES` setting, which is a dict whose keys are the -middleware class path and their values are the middleware orders. +middleware class paths and whose values are the middleware orders. Here's an example: @@ -43,10 +43,10 @@ you want to insert the middleware. The order does matter because each middleware performs a different action and your middleware could depend on some previous (or subsequent) middleware being applied. -If you want to disable a builtin middleware (the ones defined in -:setting:`SPIDER_MIDDLEWARES_BASE`, and enabled by default) you must define it -in your project :setting:`SPIDER_MIDDLEWARES` setting and assign ``None`` as its -value. For example, if you want to disable the referer middleware: +If you want to disable a built-in middleware (the ones defined in +:setting:`SPIDER_MIDDLEWARES_BASE` and enabled by default) you must define it +in your project :setting:`SPIDER_MIDDLEWARES` setting and assign ``None`` as +its value. For example, if you want to disable the referer middleware: .. code-block:: python diff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rst index 95c80d5dc..1b1a713c2 100644 --- a/docs/topics/spiders.rst +++ b/docs/topics/spiders.rst @@ -62,7 +62,7 @@ scrapy.Spider .. versionchanged:: VERSION Changes to this attribute during a crawl are now taken into account. - Let's say your target url is ``https://www.example.com/1.html``, + Let's say your target URL is ``https://www.example.com/1.html``, then add ``'example.com'`` to the list. You may modify this attribute while the spider runs, e.g. to allow @@ -87,8 +87,8 @@ scrapy.Spider :class:`~scrapy.crawler.Crawler` object to which this spider instance is bound. - Crawlers encapsulate a lot of components in the project for their single - entry access (such as extensions, middlewares, signals managers, etc). + Crawlers encapsulate a lot of components in the project for single-entry + access (such as extensions, middlewares, signal managers, etc). See :ref:`topics-api-crawler` to know more about them. .. attribute:: settings @@ -292,9 +292,8 @@ Spiders can access arguments in their `__init__` methods: self.start_urls = [f"http://www.example.com/categories/{category}"] # ... -The default `__init__` method will take any spider arguments -and copy them to the spider as attributes. -The above example can also be written as follows: +The default ``__init__`` method will take any spider arguments and copy them to +the spider as attributes. The above example can also be written as follows: .. code-block:: python @@ -318,15 +317,13 @@ specify spider arguments when calling process = CrawlerProcess() process.crawl(MySpider, category="electronics") -Keep in mind that spider arguments are only strings. -The spider will not do any parsing on its own. -If you were to set the ``start_urls`` attribute from the command line, -you would have to parse it on your own into a list -using something like :func:`ast.literal_eval` or :func:`json.loads` -and then set it as an attribute. -Otherwise, you would cause iteration over a ``start_urls`` string -(a very common python pitfall) -resulting in each character being seen as a separate url. +Keep in mind that spider arguments are only strings. The spider will not do any +parsing on its own. If you were to set the ``start_urls`` attribute from the +command line, you would have to parse it on your own into a list using +something like :func:`ast.literal_eval` or :func:`json.loads` and then set it +as an attribute. Otherwise, you would cause iteration over a ``start_urls`` +string (a very common Python pitfall) resulting in each character being seen as +a separate url. Spider arguments can also be passed through the Scrapyd ``schedule.json`` API. See `Scrapyd documentation`_. diff --git a/docs/topics/stats.rst b/docs/topics/stats.rst index 6fb7783d1..5175eb844 100644 --- a/docs/topics/stats.rst +++ b/docs/topics/stats.rst @@ -5,10 +5,11 @@ Stats Collection ================ Scrapy provides a convenient facility for collecting stats in the form of -key/values, where values are often counters. The facility is called the Stats -Collector, and can be accessed through the :attr:`~scrapy.crawler.Crawler.stats` -attribute of the :ref:`topics-api-crawler`, as illustrated by the examples in -the :ref:`topics-stats-usecases` section below. +key/value pairs, where values are often counters. The facility is called the +Stats Collector, and can be accessed through the +:attr:`~scrapy.crawler.Crawler.stats` attribute of the +:ref:`topics-api-crawler`, as illustrated by the examples in the +:ref:`topics-stats-usecases` section below. The Stats Collector API is always available, so you can always use it (to increment or set new stat keys), regardless @@ -19,7 +20,7 @@ collecting stats in your spider, Scrapy extension, or whatever code you're using the Stats Collector from. Another feature of the Stats Collector is that it's very efficient (when -enabled) and extremely efficient (almost unnoticeable) when disabled. +enabled) and almost unnoticeable when disabled. See :ref:`topics-stats-reference` below for the stats that Scrapy sets. @@ -29,7 +30,7 @@ Common Stats Collector uses =========================== Access the stats collector through the :attr:`~scrapy.crawler.Crawler.stats` -attribute. Here is an example of an extension that access stats: +attribute. Here is an example of an extension that accesses stats: .. code-block:: python diff --git a/docs/topics/telnetconsole.rst b/docs/topics/telnetconsole.rst index 54379b4b9..45d95da84 100644 --- a/docs/topics/telnetconsole.rst +++ b/docs/topics/telnetconsole.rst @@ -7,7 +7,7 @@ Telnet Console ============== Scrapy comes with a built-in telnet console for inspecting and controlling a -Scrapy running process. The telnet console is just a regular python shell +Scrapy running process. The telnet console is just a regular Python shell running inside the Scrapy process, so you can do literally anything from it. The telnet console is a :ref:`built-in Scrapy extension diff --git a/docs/versioning.rst b/docs/versioning.rst index e44496799..a483fdaa4 100644 --- a/docs/versioning.rst +++ b/docs/versioning.rst @@ -12,8 +12,8 @@ There are 3 numbers in a Scrapy version: *A.B.C* * *A* is the major version. This will rarely change and will signify very large changes. * *B* is the release number. This will include many changes including features - and things that possibly break backward compatibility, although we strive to - keep these cases at a minimum. + and things that may break backward compatibility, although we strive to keep + these cases to a minimum. * *C* is the bugfix release number. Backward-incompatibilities are explicitly mentioned in the :ref:`release notes `, diff --git a/scrapy/exceptions.py b/scrapy/exceptions.py index cd2560df1..33739e711 100644 --- a/scrapy/exceptions.py +++ b/scrapy/exceptions.py @@ -91,7 +91,7 @@ class StopDownload(Exception): exception, which is in turn stored as the ``value`` attribute of the received :class:`~twisted.python.failure.Failure` object. This means that in an errback defined as ``def errback(self, failure)``, the response can - be accessed though ``failure.value.response``. + be accessed through ``failure.value.response``. * If ``fail=False``, the request callback is called instead. diff --git a/scrapy/linkextractors/lxmlhtml.py b/scrapy/linkextractors/lxmlhtml.py index 75f9753e7..55f8a84c6 100644 --- a/scrapy/linkextractors/lxmlhtml.py +++ b/scrapy/linkextractors/lxmlhtml.py @@ -178,22 +178,22 @@ class LxmlLinkExtractor: options. It is implemented using lxml's robust HTMLParser. :param allow: a single regular expression (or list of regular expressions) - that the (absolute) urls must match in order to be extracted. If not + that the (absolute) URLs must match in order to be extracted. If not given (or empty), it will match all links. :type allow: str or list :param deny: a single regular expression (or list of regular expressions) - that the (absolute) urls must match in order to be excluded (i.e. not + that the (absolute) URLs must match in order to be excluded (i.e. not extracted). It has precedence over the ``allow`` parameter. If not - given (or empty) it won't exclude any links. + given (or empty), it won't exclude any links. :type deny: str or list - :param allow_domains: a single value or a list of string containing - domains which will be considered for extracting the links + :param allow_domains: a single value or a list of strings containing + domains that will be considered for extracting the links. :type allow_domains: str or list :param deny_domains: a single value or a list of strings containing - domains which won't be considered for extracting the links + domains that won't be considered for extracting the links. :type deny_domains: str or list :param deny_extensions: a single value or list of strings containing @@ -202,15 +202,14 @@ class LxmlLinkExtractor: :data:`scrapy.linkextractors.IGNORED_EXTENSIONS`. :type deny_extensions: list - :param restrict_xpaths: is an XPath (or list of XPath's) which defines - regions inside the response where links should be extracted from. - If given, only the text selected by those XPath will be scanned for - links. + :param restrict_xpaths: an XPath (or list of XPaths) that defines regions + inside the response where links should be extracted. If given, only the + text selected by those XPaths will be scanned for links. :type restrict_xpaths: str or list - :param restrict_css: a CSS selector (or list of selectors) which defines - regions inside the response where links should be extracted from. - Has the same behaviour as ``restrict_xpaths``. + :param restrict_css: a CSS selector (or list of selectors) that defines + regions inside the response where links should be extracted. It has the + same behaviour as ``restrict_xpaths``. :type restrict_css: str or list :param restrict_text: a single regular expression (or list of regular @@ -224,7 +223,7 @@ class LxmlLinkExtractor: Defaults to ``('a', 'area')``. Use ``'*'`` to consider every tag. :type tags: str or list - :param attrs: an attribute or list of attributes which should be considered + :param attrs: an attribute or list of attributes that should be considered when looking for links to extract (only for those tags specified in the ``tags`` parameter). Defaults to ``('href',)``. Use ``'*'`` to consider every attribute. @@ -247,20 +246,20 @@ class LxmlLinkExtractor: .. versionadded:: 2.17.0 :type deny_attrs: str or list - :param canonicalize: canonicalize each extracted url (using - w3lib.url.canonicalize_url). Defaults to ``False``. - Note that canonicalize_url is meant for duplicate checking; - it can change the URL visible at server side, so the response can be - different for requests with canonicalized and raw URLs. If you're - using LinkExtractor to follow links it is more robust to - keep the default ``canonicalize=False``. + :param canonicalize: canonicalize each extracted URL (using + w3lib.url.canonicalize_url). Defaults to ``False``. Note that + canonicalize_url is meant for duplicate checking; it can change the URL + visible at the server side, so the response can be different for + requests with canonicalized and raw URLs. If you're using LinkExtractor + to follow links, it is more robust to keep the default + ``canonicalize=False``. :type canonicalize: bool :param unique: whether duplicate filtering should be applied to extracted links. :type unique: bool - :param process_value: a function which receives each value extracted from + :param process_value: a function that receives each value extracted from the tag and attributes scanned and can modify the value and return a new one, or return ``None`` to ignore the link altogether. If not given, ``process_value`` defaults to ``lambda x: x``. @@ -291,12 +290,13 @@ class LxmlLinkExtractor: :type process_value: collections.abc.Callable :param strip: whether to strip whitespaces from extracted attributes. - According to HTML5 standard, leading and trailing whitespaces - must be stripped from ``href`` attributes of ````, ```` - and many other elements, ``src`` attribute of ````, ``