diff --git a/docs/topics/addons.rst b/docs/topics/addons.rst index 8ec7b0295..17e3c177a 100644 --- a/docs/topics/addons.rst +++ b/docs/topics/addons.rst @@ -32,7 +32,8 @@ This is an example where two add-ons are enabled in a project's Writing your own add-ons ======================== -Add-ons are Python classes that include one or both of the following methods: +Add-ons are :ref:`components ` that include one or both of +the following methods: .. method:: update_settings(settings) @@ -54,20 +55,6 @@ Add-ons are Python classes that include one or both of the following methods: :param settings: The settings object storing Scrapy/component configuration :type settings: :class:`~scrapy.settings.BaseSettings` -They can also have the following method: - -.. classmethod:: from_crawler(cls, crawler) - :noindex: - - If present, this class method is called to create an add-on instance - from a :class:`~scrapy.crawler.Crawler`. It must return a new instance - of the add-on. The crawler object provides access to all Scrapy core - components like settings and signals; it is a way for the add-on to access - them and hook its functionality into Scrapy. - - :param crawler: The crawler that uses this add-on - :type crawler: :class:`~scrapy.crawler.Crawler` - The settings set by the add-on should use the ``addon`` priority (see :ref:`populating-settings` and :func:`scrapy.settings.BaseSettings.set`):: diff --git a/docs/topics/api.rst b/docs/topics/api.rst index edc625be8..5a00fd570 100644 --- a/docs/topics/api.rst +++ b/docs/topics/api.rst @@ -12,10 +12,11 @@ extensions and middlewares. Crawler API =========== -The main entry point to Scrapy API is the :class:`~scrapy.crawler.Crawler` -object, passed to extensions through the ``from_crawler`` class method. This -object provides access to all Scrapy core components, and it's the only way for -extensions to access them and hook their functionality into Scrapy. +The main entry point to the Scrapy API is the :class:`~scrapy.crawler.Crawler` +object, which :ref:`components ` can :ref:`get for +initialization `. It provides access to all Scrapy core +components, and it is the only way for components to access them and hook their +functionality into Scrapy. .. module:: scrapy.crawler :synopsis: The Scrapy crawler diff --git a/docs/topics/components.rst b/docs/topics/components.rst index d34b3884b..3a7644379 100644 --- a/docs/topics/components.rst +++ b/docs/topics/components.rst @@ -9,6 +9,8 @@ A Scrapy component is any class whose objects are built using That includes the classes that you may assign to the following settings: +- :setting:`ADDONS` + - :setting:`DNS_RESOLVER` - :setting:`DOWNLOAD_HANDLERS` @@ -41,10 +43,80 @@ Third-party Scrapy components may also let you define additional Scrapy components, usually configurable through :ref:`settings `, to modify their behavior. +.. _from-crawler: + +Initializing from the crawler +============================= + +Any Scrapy component may optionally define the following class method: + +.. classmethod:: from_crawler(cls, crawler: scrapy.crawler.Crawler, *args, **kwargs) + + Return an instance of the component based on *crawler*. + + *args* and *kwargs* are component-specific arguments that some components + receive. However, most components do not get any arguments, and instead + :ref:`use settings `. + + If a component class defines this method, this class method is called to + create any instance of the component. + + The *crawler* object provides access to all Scrapy core components like + :ref:`settings ` and :ref:`signals `, + allowing the component to access them and hook its functionality into + Scrapy. + +.. _component-settings: + +Settings +======== + +Components can be configured through :ref:`settings `. + +Components can read any setting from the +:attr:`~scrapy.crawler.Crawler.settings` attribute of the +:class:`~scrapy.crawler.Crawler` object they can :ref:`get for initialization +`. That includes both built-in and custom settings. + +For example: + +.. code-block:: python + + class MyExtension: + @classmethod + def from_crawler(cls, crawler): + settings = crawler.settings + return cls(settings.getbool("LOG_ENABLED")) + + def __init__(self, log_is_enabled=False): + if log_is_enabled: + print("log is enabled!") + +Components do not need to declare their custom settings programmatically. +However, they should document them, so that users know they exist and how to +use them. + +It is a good practice to prefix custom settings with the name of the component, +to avoid collisions with custom settings of other existing (or future) +components. For example, an extension called ``WarcCaching`` could prefix its +custom settings with ``WARC_CACHING_``. + +Another good practice, mainly for components meant for :ref:`component priority +dictionaries `, is to provide a boolean setting +called ``_ENABLED`` (e.g. ``WARC_CACHING_ENABLED``) to allow toggling +that component on and off without changing the component priority dictionary +setting. You can usually check the value of such a setting during +initialization, and if ``False``, raise +:exc:`~scrapy.exceptions.NotConfigured`. + +When choosing a name for a custom setting, it is also a good idea to have a +look at the names of :ref:`built-in settings `, to try to +maintain consistency with them. + .. _enforce-component-requirements: -Enforcing component requirements -================================ +Enforcing requirements +====================== Sometimes, your components may only be intended to work under certain conditions. For example, they may require a minimum version of Scrapy to work as @@ -58,8 +130,8 @@ In the case of :ref:`downloader middlewares `, :ref:`extensions `, :ref:`item pipelines `, and :ref:`spider middlewares `, you should raise -:exc:`scrapy.exceptions.NotConfigured`, passing a description of the issue as a -parameter to the exception so that it is printed in the logs, for the user to +:exc:`~scrapy.exceptions.NotConfigured`, passing a description of the issue as +a parameter to the exception so that it is printed in the logs, for the user to see. For other components, feel free to raise whatever other exception feels right to you; for example, :exc:`RuntimeError` would make sense for a Scrapy version mismatch, while :exc:`ValueError` may be better if the issue is the diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst index ab7e6a0ec..60b6aab78 100644 --- a/docs/topics/downloader-middleware.rst +++ b/docs/topics/downloader-middleware.rst @@ -61,12 +61,8 @@ particular setting. See each middleware documentation for more info. Writing your own downloader middleware ====================================== -Each downloader middleware is a Python class that defines one or more of the -methods defined below. - -The main entry point is the ``from_crawler`` class method, which receives a -:class:`~scrapy.crawler.Crawler` instance. The :class:`~scrapy.crawler.Crawler` -object gives you access, for example, to the :ref:`settings `. +Each downloader middleware is a :ref:`component ` that +defines one or more of these methods: .. module:: scrapy.downloadermiddlewares @@ -167,17 +163,6 @@ object gives you access, for example, to the :ref:`settings `. :param spider: the spider for which this request is intended :type spider: :class:`~scrapy.Spider` object - .. method:: from_crawler(cls, crawler) - - If present, this classmethod is called to create a middleware instance - from a :class:`~scrapy.crawler.Crawler`. It must return a new instance - of the middleware. Crawler object provides access to all Scrapy core - components like settings and signals; it is a way for middleware to - access them and hook its functionality into Scrapy. - - :param crawler: crawler that uses this middleware - :type crawler: :class:`~scrapy.crawler.Crawler` object - .. _topics-downloader-middleware-ref: Built-in downloader middleware reference diff --git a/docs/topics/email.rst b/docs/topics/email.rst index 8f7a2357a..1d7bad787 100644 --- a/docs/topics/email.rst +++ b/docs/topics/email.rst @@ -50,9 +50,9 @@ And here is how to use it to send an e-mail (without attachments): MailSender class reference ========================== -MailSender is the preferred class to use for sending emails from Scrapy, as it -uses :doc:`Twisted non-blocking IO `, like the -rest of the framework. +The MailSender :ref:`components ` is the preferred class to +use for sending emails from Scrapy, as it uses :doc:`Twisted non-blocking IO +`, like the rest of the framework. .. class:: MailSender(smtphost=None, mailfrom=None, smtpuser=None, smtppass=None, smtpport=None) @@ -81,14 +81,6 @@ rest of the framework. :param smtpssl: enforce using a secure SSL connection :type smtpssl: bool - .. classmethod:: from_crawler(crawler) - - Instantiate using a :class:`scrapy.Crawler` instance, which will - respect :ref:`these Scrapy settings `. - - :param crawler: the crawler - :type settings: :class:`scrapy.Crawler` object - .. method:: send(to, subject, body, cc=None, attachs=(), mimetype='text/plain', charset=None) Send email to the given recipients. diff --git a/docs/topics/extensions.rst b/docs/topics/extensions.rst index 23bbcfcb5..e1e3dd6b4 100644 --- a/docs/topics/extensions.rst +++ b/docs/topics/extensions.rst @@ -4,34 +4,21 @@ Extensions ========== -The extensions framework provides a mechanism for inserting your own -custom functionality into Scrapy. +Extensions are :ref:`components ` that allow inserting your +own custom functionality into Scrapy. -Extensions are just regular classes. +Unlike other components, extensions do not have a specific role in Scrapy. They +are “wildcard” components that can be used for anything that does not fit the +role of any other type of component. -Extension settings -================== +Loading and activating extensions +================================= -Extensions use the :ref:`Scrapy settings ` to manage their -settings, just like any other Scrapy code. +Extensions are loaded at startup by creating a single instance of the extension +class per spider being run. -It is customary for extensions to prefix their settings with their own name, to -avoid collision with existing (and future) extensions. For example, a -hypothetical extension to handle `Google Sitemaps`_ would use settings like -``GOOGLESITEMAP_ENABLED``, ``GOOGLESITEMAP_DEPTH``, and so on. - -.. _Google Sitemaps: https://en.wikipedia.org/wiki/Sitemaps - -Loading & activating extensions -=============================== - -Extensions are loaded and activated at startup by instantiating a single -instance of the extension class per spider being run. All the extension -initialization code must be performed in the class ``__init__`` method. - -To make an extension available, add it to the :setting:`EXTENSIONS` setting in -your Scrapy settings. In :setting:`EXTENSIONS`, each extension is represented -by a string: the full Python path to the extension's class name. For example: +To enable an extension, add it to the :setting:`EXTENSIONS` setting. For +example: .. code-block:: python @@ -40,55 +27,24 @@ by a string: the full Python path to the extension's class name. For example: "scrapy.extensions.telnet.TelnetConsole": 500, } - -As you can see, the :setting:`EXTENSIONS` setting is a dict where the keys are -the extension paths, and their values are the orders, which define the -extension *loading* order. The :setting:`EXTENSIONS` setting is merged with the -:setting:`EXTENSIONS_BASE` setting defined in Scrapy (and not meant to be -overridden) and then sorted by order to get the final sorted list of enabled -extensions. +:setting:`EXTENSIONS` is merged with :setting:`EXTENSIONS_BASE` (not meant to +be overridden), and the priorities in the resulting value determine the +*loading* order. As extensions typically do not depend on each other, their loading order is irrelevant in most cases. This is why the :setting:`EXTENSIONS_BASE` setting -defines all extensions with the same order (``0``). However, this feature can -be exploited if you need to add an extension which depends on other extensions -already loaded. - -Available, enabled and disabled extensions -========================================== - -Not all available extensions will be enabled. Some of them usually depend on a -particular setting. For example, the HTTP Cache extension is available by default -but disabled unless the :setting:`HTTPCACHE_ENABLED` setting is set. - -Disabling an extension -====================== - -In order to disable an extension that comes enabled by default (i.e. those -included in the :setting:`EXTENSIONS_BASE` setting) you must set its order to -``None``. For example: - -.. code-block:: python - - EXTENSIONS = { - "scrapy.extensions.corestats.CoreStats": None, - } +defines all extensions with the same order (``0``). However, you may need to +carefully use priorities if you add an extension that depends on other +extensions being already loaded. Writing your own extension ========================== -Each extension is a Python class. The main entry point for a Scrapy extension -(this also includes middlewares and pipelines) is the ``from_crawler`` -class method which receives a ``Crawler`` instance. Through the Crawler object -you can access settings, signals, stats, and also control the crawling behaviour. +Each extension is a :ref:`component `. Typically, extensions connect to :ref:`signals ` and perform tasks triggered by them. -Finally, if the ``from_crawler`` method raises the -:exc:`~scrapy.exceptions.NotConfigured` exception, the extension will be -disabled. Otherwise, the extension will be enabled. - Sample extension ---------------- diff --git a/docs/topics/item-pipeline.rst b/docs/topics/item-pipeline.rst index 310f153e8..dc27ce6ca 100644 --- a/docs/topics/item-pipeline.rst +++ b/docs/topics/item-pipeline.rst @@ -23,7 +23,8 @@ Typical uses of item pipelines are: Writing your own item pipeline ============================== -Each item pipeline component is a Python class that must implement the following method: +Each item pipeline is a :ref:`component ` that must +implement the following method: .. method:: process_item(self, item, spider) @@ -60,17 +61,6 @@ Additionally, they may also implement the following methods: :param spider: the spider which was closed :type spider: :class:`~scrapy.Spider` object -.. classmethod:: from_crawler(cls, crawler) - - If present, this class method is called to create a pipeline instance - from a :class:`~scrapy.crawler.Crawler`. It must return a new instance - of the pipeline. Crawler object provides access to all Scrapy core - components like settings and signals; it is a way for pipeline to - access them and hook its functionality into Scrapy. - - :param crawler: crawler that uses this pipeline - :type crawler: :class:`~scrapy.crawler.Crawler` object - Item pipeline example ===================== @@ -139,8 +129,8 @@ In this example we'll write items to MongoDB_ using pymongo_. MongoDB address and database name are specified in Scrapy settings; MongoDB collection is named after item class. -The main point of this example is to show how to use :meth:`from_crawler` -method and how to clean up the resources properly. +The main point of this example is to show how to :ref:`get the crawler +` and how to clean up the resources properly. .. skip: next .. code-block:: python diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rst index 0375e0ff1..77837378e 100644 --- a/docs/topics/request-response.rst +++ b/docs/topics/request-response.rst @@ -463,35 +463,17 @@ import path. Writing your own request fingerprinter ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -A request fingerprinter is a class that must implement the following method: +A request fingerprinter is a :ref:`component ` that must +implement the following method: .. currentmodule:: None -.. method:: fingerprint(self, request) +.. method:: fingerprint(self, request: scrapy.Request) Return a :class:`bytes` object that uniquely identifies *request*. See also :ref:`request-fingerprint-restrictions`. - :param request: request to fingerprint - :type request: scrapy.Request - -Additionally, it may also implement the following method: - -.. classmethod:: from_crawler(cls, crawler) - :noindex: - - If present, this class method is called to create a request fingerprinter - instance from a :class:`~scrapy.crawler.Crawler` object. It must return a - new instance of the request fingerprinter. - - *crawler* provides access to all Scrapy core components like settings and - signals; it is a way for the request fingerprinter to access them and hook - its functionality into Scrapy. - - :param crawler: crawler that uses this request fingerprinter - :type crawler: :class:`~scrapy.crawler.Crawler` object - .. currentmodule:: scrapy.http The :meth:`fingerprint` method of the default request fingerprinter, diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst index 7646aca4f..b2bb7148f 100644 --- a/docs/topics/settings.rst +++ b/docs/topics/settings.rst @@ -204,7 +204,7 @@ How to access settings .. highlight:: python -In a spider, the settings are available through ``self.settings``: +In a spider, settings are available through ``self.settings``: .. code-block:: python @@ -217,37 +217,17 @@ In a spider, the settings are available through ``self.settings``: .. note:: The ``settings`` attribute is set in the base Spider class after the spider - is initialized. If you want to use the settings before the initialization + is initialized. If you want to use settings before the initialization (e.g., in your spider's ``__init__()`` method), you'll need to override the :meth:`~scrapy.Spider.from_crawler` method. -Settings can be accessed through the :attr:`scrapy.crawler.Crawler.settings` -attribute of the Crawler that is passed to ``from_crawler`` method in -extensions, middlewares and item pipelines: +:ref:`Components ` can also :ref:`access settings +`. -.. code-block:: python - - class MyExtension: - def __init__(self, log_is_enabled=False): - if log_is_enabled: - print("log is enabled!") - - @classmethod - def from_crawler(cls, crawler): - settings = crawler.settings - return cls(settings.getbool("LOG_ENABLED")) - -The settings object can be used like a dict (e.g., -``settings['LOG_ENABLED']``), but it's usually preferred to extract the setting -in the format you need it to avoid type errors, using one of the methods -provided by the :class:`~scrapy.settings.Settings` API. - -Rationale for setting names -=========================== - -Setting names are usually prefixed with the component that they configure. For -example, proper setting names for a fictional robots.txt extension would be -``ROBOTSTXT_ENABLED``, ``ROBOTSTXT_OBEY``, ``ROBOTSTXT_CACHEDIR``, etc. +The ``settings`` object can be used like a :class:`dict` (e.g. +``settings["LOG_ENABLED"]``). However, to support non-string setting values, +which may be passed from the command line as strings, it is recommended to use +one of the methods provided by the :class:`~scrapy.settings.Settings` API. .. _component-priority-dictionaries: @@ -1211,7 +1191,8 @@ EXTENSIONS Default:: ``{}`` -A dict containing the extensions enabled in your project, and their orders. +:ref:`Component priority dictionary ` of +enabled extensions. See :ref:`topics-extensions`. .. setting:: EXTENSIONS_BASE diff --git a/docs/topics/spider-middleware.rst b/docs/topics/spider-middleware.rst index 2b59cabe1..567a875b6 100644 --- a/docs/topics/spider-middleware.rst +++ b/docs/topics/spider-middleware.rst @@ -63,17 +63,38 @@ particular setting. See each middleware documentation for more info. Writing your own spider middleware ================================== -Each spider middleware is a Python class that defines one or more of the -methods defined below. - -The main entry point is the ``from_crawler`` class method, which receives a -:class:`~scrapy.crawler.Crawler` instance. The :class:`~scrapy.crawler.Crawler` -object gives you access, for example, to the :ref:`settings `. +Each spider middleware is a :ref:`component ` that defines +one or more of these methods: .. module:: scrapy.spidermiddlewares .. class:: SpiderMiddleware + .. method:: process_start_requests(start_requests, spider) + + This method is called with the start requests of the spider, and works + similarly to the :meth:`process_spider_output` method, except that it + doesn't have a response associated and must return only requests (not + items). + + It receives an iterable (in the ``start_requests`` parameter) and must + return another iterable of :class:`~scrapy.Request` objects and/or :ref:`item objects `. + + .. note:: When implementing this method in your spider middleware, you + should always return an iterable (that follows the input one) and + not consume all ``start_requests`` iterator because it can be very + large (or even unbounded) and cause a memory overflow. The Scrapy + engine is designed to pull start requests while it has capacity to + process them, so the start requests iterator can be effectively + endless where there is some other condition for stopping the spider + (like a time limit or item/page count). + + :param start_requests: the start requests + :type start_requests: an iterable of :class:`~scrapy.Request` + + :param spider: the spider to whom the start requests belong + :type spider: :class:`~scrapy.Spider` object + .. method:: process_spider_input(response, spider) This method is called for each response that goes through the spider @@ -168,42 +189,6 @@ object gives you access, for example, to the :ref:`settings `. :param spider: the spider which raised the exception :type spider: :class:`~scrapy.Spider` object - .. method:: process_start_requests(start_requests, spider) - - This method is called with the start requests of the spider, and works - similarly to the :meth:`process_spider_output` method, except that it - doesn't have a response associated and must return only requests (not - items). - - It receives an iterable (in the ``start_requests`` parameter) and must - return another iterable of :class:`~scrapy.Request` objects and/or :ref:`item objects `. - - .. note:: When implementing this method in your spider middleware, you - should always return an iterable (that follows the input one) and - not consume all ``start_requests`` iterator because it can be very - large (or even unbounded) and cause a memory overflow. The Scrapy - engine is designed to pull start requests while it has capacity to - process them, so the start requests iterator can be effectively - endless where there is some other condition for stopping the spider - (like a time limit or item/page count). - - :param start_requests: the start requests - :type start_requests: an iterable of :class:`~scrapy.Request` - - :param spider: the spider to whom the start requests belong - :type spider: :class:`~scrapy.Spider` object - - .. method:: from_crawler(cls, crawler) - - If present, this classmethod is called to create a middleware instance - from a :class:`~scrapy.crawler.Crawler`. It must return a new instance - of the middleware. Crawler object provides access to all Scrapy core - components like settings and signals; it is a way for middleware to - access them and hook its functionality into Scrapy. - - :param crawler: crawler that uses this middleware - :type crawler: :class:`~scrapy.crawler.Crawler` object - .. _topics-spider-middleware-ref: Built-in spider middleware reference