mirror of https://github.com/scrapy/scrapy.git
Centralize from_crawler docs (and somewhat related changes) (#6723)
This commit is contained in:
Adrián Chaves
GitHub
parent
5a0690c89d
commit
ba28d96d3e
|
|
@ -32,7 +32,8 @@ This is an example where two add-ons are enabled in a project's
|
|||
Writing your own add-ons
|
||||
========================
|
||||
|
||||
Add-ons are Python classes that include one or both of the following methods:
|
||||
Add-ons are :ref:`components <topics-components>` that include one or both of
|
||||
the following methods:
|
||||
|
||||
.. method:: update_settings(settings)
|
||||
|
||||
|
|
@ -54,20 +55,6 @@ Add-ons are Python classes that include one or both of the following methods:
|
|||
:param settings: The settings object storing Scrapy/component configuration
|
||||
:type settings: :class:`~scrapy.settings.BaseSettings`
|
||||
|
||||
They can also have the following method:
|
||||
|
||||
.. classmethod:: from_crawler(cls, crawler)
|
||||
:noindex:
|
||||
|
||||
If present, this class method is called to create an add-on instance
|
||||
from a :class:`~scrapy.crawler.Crawler`. It must return a new instance
|
||||
of the add-on. The crawler object provides access to all Scrapy core
|
||||
components like settings and signals; it is a way for the add-on to access
|
||||
them and hook its functionality into Scrapy.
|
||||
|
||||
:param crawler: The crawler that uses this add-on
|
||||
:type crawler: :class:`~scrapy.crawler.Crawler`
|
||||
|
||||
The settings set by the add-on should use the ``addon`` priority (see
|
||||
:ref:`populating-settings` and :func:`scrapy.settings.BaseSettings.set`)::
|
||||
|
||||
|
|
|
|||
|
|
@ -12,10 +12,11 @@ extensions and middlewares.
|
|||
Crawler API
|
||||
===========
|
||||
|
||||
The main entry point to Scrapy API is the :class:`~scrapy.crawler.Crawler`
|
||||
object, passed to extensions through the ``from_crawler`` class method. This
|
||||
object provides access to all Scrapy core components, and it's the only way for
|
||||
extensions to access them and hook their functionality into Scrapy.
|
||||
The main entry point to the Scrapy API is the :class:`~scrapy.crawler.Crawler`
|
||||
object, which :ref:`components <topics-components>` can :ref:`get for
|
||||
initialization <from-crawler>`. It provides access to all Scrapy core
|
||||
components, and it is the only way for components to access them and hook their
|
||||
functionality into Scrapy.
|
||||
|
||||
.. module:: scrapy.crawler
|
||||
:synopsis: The Scrapy crawler
|
||||
|
|
|
|||
|
|
@ -9,6 +9,8 @@ A Scrapy component is any class whose objects are built using
|
|||
|
||||
That includes the classes that you may assign to the following settings:
|
||||
|
||||
- :setting:`ADDONS`
|
||||
|
||||
- :setting:`DNS_RESOLVER`
|
||||
|
||||
- :setting:`DOWNLOAD_HANDLERS`
|
||||
|
|
@ -41,10 +43,80 @@ Third-party Scrapy components may also let you define additional Scrapy
|
|||
components, usually configurable through :ref:`settings <topics-settings>`, to
|
||||
modify their behavior.
|
||||
|
||||
.. _from-crawler:
|
||||
|
||||
Initializing from the crawler
|
||||
=============================
|
||||
|
||||
Any Scrapy component may optionally define the following class method:
|
||||
|
||||
.. classmethod:: from_crawler(cls, crawler: scrapy.crawler.Crawler, *args, **kwargs)
|
||||
|
||||
Return an instance of the component based on *crawler*.
|
||||
|
||||
*args* and *kwargs* are component-specific arguments that some components
|
||||
receive. However, most components do not get any arguments, and instead
|
||||
:ref:`use settings <component-settings>`.
|
||||
|
||||
If a component class defines this method, this class method is called to
|
||||
create any instance of the component.
|
||||
|
||||
The *crawler* object provides access to all Scrapy core components like
|
||||
:ref:`settings <topics-settings>` and :ref:`signals <topics-signals>`,
|
||||
allowing the component to access them and hook its functionality into
|
||||
Scrapy.
|
||||
|
||||
.. _component-settings:
|
||||
|
||||
Settings
|
||||
========
|
||||
|
||||
Components can be configured through :ref:`settings <topics-settings>`.
|
||||
|
||||
Components can read any setting from the
|
||||
:attr:`~scrapy.crawler.Crawler.settings` attribute of the
|
||||
:class:`~scrapy.crawler.Crawler` object they can :ref:`get for initialization
|
||||
<from-crawler>`. That includes both built-in and custom settings.
|
||||
|
||||
For example:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
class MyExtension:
|
||||
@classmethod
|
||||
def from_crawler(cls, crawler):
|
||||
settings = crawler.settings
|
||||
return cls(settings.getbool("LOG_ENABLED"))
|
||||
|
||||
def __init__(self, log_is_enabled=False):
|
||||
if log_is_enabled:
|
||||
print("log is enabled!")
|
||||
|
||||
Components do not need to declare their custom settings programmatically.
|
||||
However, they should document them, so that users know they exist and how to
|
||||
use them.
|
||||
|
||||
It is a good practice to prefix custom settings with the name of the component,
|
||||
to avoid collisions with custom settings of other existing (or future)
|
||||
components. For example, an extension called ``WarcCaching`` could prefix its
|
||||
custom settings with ``WARC_CACHING_``.
|
||||
|
||||
Another good practice, mainly for components meant for :ref:`component priority
|
||||
dictionaries <component-priority-dictionaries>`, is to provide a boolean setting
|
||||
called ``<PREFIX>_ENABLED`` (e.g. ``WARC_CACHING_ENABLED``) to allow toggling
|
||||
that component on and off without changing the component priority dictionary
|
||||
setting. You can usually check the value of such a setting during
|
||||
initialization, and if ``False``, raise
|
||||
:exc:`~scrapy.exceptions.NotConfigured`.
|
||||
|
||||
When choosing a name for a custom setting, it is also a good idea to have a
|
||||
look at the names of :ref:`built-in settings <topics-settings-ref>`, to try to
|
||||
maintain consistency with them.
|
||||
|
||||
.. _enforce-component-requirements:
|
||||
|
||||
Enforcing component requirements
|
||||
================================
|
||||
Enforcing requirements
|
||||
======================
|
||||
|
||||
Sometimes, your components may only be intended to work under certain
|
||||
conditions. For example, they may require a minimum version of Scrapy to work as
|
||||
|
|
@ -58,8 +130,8 @@ In the case of :ref:`downloader middlewares <topics-downloader-middleware>`,
|
|||
:ref:`extensions <topics-extensions>`, :ref:`item pipelines
|
||||
<topics-item-pipeline>`, and :ref:`spider middlewares
|
||||
<topics-spider-middleware>`, you should raise
|
||||
:exc:`scrapy.exceptions.NotConfigured`, passing a description of the issue as a
|
||||
parameter to the exception so that it is printed in the logs, for the user to
|
||||
:exc:`~scrapy.exceptions.NotConfigured`, passing a description of the issue as
|
||||
a parameter to the exception so that it is printed in the logs, for the user to
|
||||
see. For other components, feel free to raise whatever other exception feels
|
||||
right to you; for example, :exc:`RuntimeError` would make sense for a Scrapy
|
||||
version mismatch, while :exc:`ValueError` may be better if the issue is the
|
||||
|
|
|
|||
|
|
@ -61,12 +61,8 @@ particular setting. See each middleware documentation for more info.
|
|||
Writing your own downloader middleware
|
||||
======================================
|
||||
|
||||
Each downloader middleware is a Python class that defines one or more of the
|
||||
methods defined below.
|
||||
|
||||
The main entry point is the ``from_crawler`` class method, which receives a
|
||||
:class:`~scrapy.crawler.Crawler` instance. The :class:`~scrapy.crawler.Crawler`
|
||||
object gives you access, for example, to the :ref:`settings <topics-settings>`.
|
||||
Each downloader middleware is a :ref:`component <topics-components>` that
|
||||
defines one or more of these methods:
|
||||
|
||||
.. module:: scrapy.downloadermiddlewares
|
||||
|
||||
|
|
@ -167,17 +163,6 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
|
|||
:param spider: the spider for which this request is intended
|
||||
:type spider: :class:`~scrapy.Spider` object
|
||||
|
||||
.. method:: from_crawler(cls, crawler)
|
||||
|
||||
If present, this classmethod is called to create a middleware instance
|
||||
from a :class:`~scrapy.crawler.Crawler`. It must return a new instance
|
||||
of the middleware. Crawler object provides access to all Scrapy core
|
||||
components like settings and signals; it is a way for middleware to
|
||||
access them and hook its functionality into Scrapy.
|
||||
|
||||
:param crawler: crawler that uses this middleware
|
||||
:type crawler: :class:`~scrapy.crawler.Crawler` object
|
||||
|
||||
.. _topics-downloader-middleware-ref:
|
||||
|
||||
Built-in downloader middleware reference
|
||||
|
|
|
|||
|
|
@ -50,9 +50,9 @@ And here is how to use it to send an e-mail (without attachments):
|
|||
MailSender class reference
|
||||
==========================
|
||||
|
||||
MailSender is the preferred class to use for sending emails from Scrapy, as it
|
||||
uses :doc:`Twisted non-blocking IO <twisted:core/howto/defer-intro>`, like the
|
||||
rest of the framework.
|
||||
The MailSender :ref:`components <topics-components>` is the preferred class to
|
||||
use for sending emails from Scrapy, as it uses :doc:`Twisted non-blocking IO
|
||||
<twisted:core/howto/defer-intro>`, like the rest of the framework.
|
||||
|
||||
.. class:: MailSender(smtphost=None, mailfrom=None, smtpuser=None, smtppass=None, smtpport=None)
|
||||
|
||||
|
|
@ -81,14 +81,6 @@ rest of the framework.
|
|||
:param smtpssl: enforce using a secure SSL connection
|
||||
:type smtpssl: bool
|
||||
|
||||
.. classmethod:: from_crawler(crawler)
|
||||
|
||||
Instantiate using a :class:`scrapy.Crawler` instance, which will
|
||||
respect :ref:`these Scrapy settings <topics-email-settings>`.
|
||||
|
||||
:param crawler: the crawler
|
||||
:type settings: :class:`scrapy.Crawler` object
|
||||
|
||||
.. method:: send(to, subject, body, cc=None, attachs=(), mimetype='text/plain', charset=None)
|
||||
|
||||
Send email to the given recipients.
|
||||
|
|
|
|||
|
|
@ -4,34 +4,21 @@
|
|||
Extensions
|
||||
==========
|
||||
|
||||
The extensions framework provides a mechanism for inserting your own
|
||||
custom functionality into Scrapy.
|
||||
Extensions are :ref:`components <topics-components>` that allow inserting your
|
||||
own custom functionality into Scrapy.
|
||||
|
||||
Extensions are just regular classes.
|
||||
Unlike other components, extensions do not have a specific role in Scrapy. They
|
||||
are “wildcard” components that can be used for anything that does not fit the
|
||||
role of any other type of component.
|
||||
|
||||
Extension settings
|
||||
==================
|
||||
Loading and activating extensions
|
||||
=================================
|
||||
|
||||
Extensions use the :ref:`Scrapy settings <topics-settings>` to manage their
|
||||
settings, just like any other Scrapy code.
|
||||
Extensions are loaded at startup by creating a single instance of the extension
|
||||
class per spider being run.
|
||||
|
||||
It is customary for extensions to prefix their settings with their own name, to
|
||||
avoid collision with existing (and future) extensions. For example, a
|
||||
hypothetical extension to handle `Google Sitemaps`_ would use settings like
|
||||
``GOOGLESITEMAP_ENABLED``, ``GOOGLESITEMAP_DEPTH``, and so on.
|
||||
|
||||
.. _Google Sitemaps: https://en.wikipedia.org/wiki/Sitemaps
|
||||
|
||||
Loading & activating extensions
|
||||
===============================
|
||||
|
||||
Extensions are loaded and activated at startup by instantiating a single
|
||||
instance of the extension class per spider being run. All the extension
|
||||
initialization code must be performed in the class ``__init__`` method.
|
||||
|
||||
To make an extension available, add it to the :setting:`EXTENSIONS` setting in
|
||||
your Scrapy settings. In :setting:`EXTENSIONS`, each extension is represented
|
||||
by a string: the full Python path to the extension's class name. For example:
|
||||
To enable an extension, add it to the :setting:`EXTENSIONS` setting. For
|
||||
example:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
@ -40,55 +27,24 @@ by a string: the full Python path to the extension's class name. For example:
|
|||
"scrapy.extensions.telnet.TelnetConsole": 500,
|
||||
}
|
||||
|
||||
|
||||
As you can see, the :setting:`EXTENSIONS` setting is a dict where the keys are
|
||||
the extension paths, and their values are the orders, which define the
|
||||
extension *loading* order. The :setting:`EXTENSIONS` setting is merged with the
|
||||
:setting:`EXTENSIONS_BASE` setting defined in Scrapy (and not meant to be
|
||||
overridden) and then sorted by order to get the final sorted list of enabled
|
||||
extensions.
|
||||
:setting:`EXTENSIONS` is merged with :setting:`EXTENSIONS_BASE` (not meant to
|
||||
be overridden), and the priorities in the resulting value determine the
|
||||
*loading* order.
|
||||
|
||||
As extensions typically do not depend on each other, their loading order is
|
||||
irrelevant in most cases. This is why the :setting:`EXTENSIONS_BASE` setting
|
||||
defines all extensions with the same order (``0``). However, this feature can
|
||||
be exploited if you need to add an extension which depends on other extensions
|
||||
already loaded.
|
||||
|
||||
Available, enabled and disabled extensions
|
||||
==========================================
|
||||
|
||||
Not all available extensions will be enabled. Some of them usually depend on a
|
||||
particular setting. For example, the HTTP Cache extension is available by default
|
||||
but disabled unless the :setting:`HTTPCACHE_ENABLED` setting is set.
|
||||
|
||||
Disabling an extension
|
||||
======================
|
||||
|
||||
In order to disable an extension that comes enabled by default (i.e. those
|
||||
included in the :setting:`EXTENSIONS_BASE` setting) you must set its order to
|
||||
``None``. For example:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
EXTENSIONS = {
|
||||
"scrapy.extensions.corestats.CoreStats": None,
|
||||
}
|
||||
defines all extensions with the same order (``0``). However, you may need to
|
||||
carefully use priorities if you add an extension that depends on other
|
||||
extensions being already loaded.
|
||||
|
||||
Writing your own extension
|
||||
==========================
|
||||
|
||||
Each extension is a Python class. The main entry point for a Scrapy extension
|
||||
(this also includes middlewares and pipelines) is the ``from_crawler``
|
||||
class method which receives a ``Crawler`` instance. Through the Crawler object
|
||||
you can access settings, signals, stats, and also control the crawling behaviour.
|
||||
Each extension is a :ref:`component <topics-components>`.
|
||||
|
||||
Typically, extensions connect to :ref:`signals <topics-signals>` and perform
|
||||
tasks triggered by them.
|
||||
|
||||
Finally, if the ``from_crawler`` method raises the
|
||||
:exc:`~scrapy.exceptions.NotConfigured` exception, the extension will be
|
||||
disabled. Otherwise, the extension will be enabled.
|
||||
|
||||
Sample extension
|
||||
----------------
|
||||
|
||||
|
|
|
|||
|
|
@ -23,7 +23,8 @@ Typical uses of item pipelines are:
|
|||
Writing your own item pipeline
|
||||
==============================
|
||||
|
||||
Each item pipeline component is a Python class that must implement the following method:
|
||||
Each item pipeline is a :ref:`component <topics-components>` that must
|
||||
implement the following method:
|
||||
|
||||
.. method:: process_item(self, item, spider)
|
||||
|
||||
|
|
@ -60,17 +61,6 @@ Additionally, they may also implement the following methods:
|
|||
:param spider: the spider which was closed
|
||||
:type spider: :class:`~scrapy.Spider` object
|
||||
|
||||
.. classmethod:: from_crawler(cls, crawler)
|
||||
|
||||
If present, this class method is called to create a pipeline instance
|
||||
from a :class:`~scrapy.crawler.Crawler`. It must return a new instance
|
||||
of the pipeline. Crawler object provides access to all Scrapy core
|
||||
components like settings and signals; it is a way for pipeline to
|
||||
access them and hook its functionality into Scrapy.
|
||||
|
||||
:param crawler: crawler that uses this pipeline
|
||||
:type crawler: :class:`~scrapy.crawler.Crawler` object
|
||||
|
||||
|
||||
Item pipeline example
|
||||
=====================
|
||||
|
|
@ -139,8 +129,8 @@ In this example we'll write items to MongoDB_ using pymongo_.
|
|||
MongoDB address and database name are specified in Scrapy settings;
|
||||
MongoDB collection is named after item class.
|
||||
|
||||
The main point of this example is to show how to use :meth:`from_crawler`
|
||||
method and how to clean up the resources properly.
|
||||
The main point of this example is to show how to :ref:`get the crawler
|
||||
<from-crawler>` and how to clean up the resources properly.
|
||||
|
||||
.. skip: next
|
||||
.. code-block:: python
|
||||
|
|
|
|||
|
|
@ -463,35 +463,17 @@ import path.
|
|||
Writing your own request fingerprinter
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
A request fingerprinter is a class that must implement the following method:
|
||||
A request fingerprinter is a :ref:`component <topics-components>` that must
|
||||
implement the following method:
|
||||
|
||||
.. currentmodule:: None
|
||||
|
||||
.. method:: fingerprint(self, request)
|
||||
.. method:: fingerprint(self, request: scrapy.Request)
|
||||
|
||||
Return a :class:`bytes` object that uniquely identifies *request*.
|
||||
|
||||
See also :ref:`request-fingerprint-restrictions`.
|
||||
|
||||
:param request: request to fingerprint
|
||||
:type request: scrapy.Request
|
||||
|
||||
Additionally, it may also implement the following method:
|
||||
|
||||
.. classmethod:: from_crawler(cls, crawler)
|
||||
:noindex:
|
||||
|
||||
If present, this class method is called to create a request fingerprinter
|
||||
instance from a :class:`~scrapy.crawler.Crawler` object. It must return a
|
||||
new instance of the request fingerprinter.
|
||||
|
||||
*crawler* provides access to all Scrapy core components like settings and
|
||||
signals; it is a way for the request fingerprinter to access them and hook
|
||||
its functionality into Scrapy.
|
||||
|
||||
:param crawler: crawler that uses this request fingerprinter
|
||||
:type crawler: :class:`~scrapy.crawler.Crawler` object
|
||||
|
||||
.. currentmodule:: scrapy.http
|
||||
|
||||
The :meth:`fingerprint` method of the default request fingerprinter,
|
||||
|
|
|
|||
|
|
@ -204,7 +204,7 @@ How to access settings
|
|||
|
||||
.. highlight:: python
|
||||
|
||||
In a spider, the settings are available through ``self.settings``:
|
||||
In a spider, settings are available through ``self.settings``:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
@ -217,37 +217,17 @@ In a spider, the settings are available through ``self.settings``:
|
|||
|
||||
.. note::
|
||||
The ``settings`` attribute is set in the base Spider class after the spider
|
||||
is initialized. If you want to use the settings before the initialization
|
||||
is initialized. If you want to use settings before the initialization
|
||||
(e.g., in your spider's ``__init__()`` method), you'll need to override the
|
||||
:meth:`~scrapy.Spider.from_crawler` method.
|
||||
|
||||
Settings can be accessed through the :attr:`scrapy.crawler.Crawler.settings`
|
||||
attribute of the Crawler that is passed to ``from_crawler`` method in
|
||||
extensions, middlewares and item pipelines:
|
||||
:ref:`Components <topics-components>` can also :ref:`access settings
|
||||
<component-settings>`.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
class MyExtension:
|
||||
def __init__(self, log_is_enabled=False):
|
||||
if log_is_enabled:
|
||||
print("log is enabled!")
|
||||
|
||||
@classmethod
|
||||
def from_crawler(cls, crawler):
|
||||
settings = crawler.settings
|
||||
return cls(settings.getbool("LOG_ENABLED"))
|
||||
|
||||
The settings object can be used like a dict (e.g.,
|
||||
``settings['LOG_ENABLED']``), but it's usually preferred to extract the setting
|
||||
in the format you need it to avoid type errors, using one of the methods
|
||||
provided by the :class:`~scrapy.settings.Settings` API.
|
||||
|
||||
Rationale for setting names
|
||||
===========================
|
||||
|
||||
Setting names are usually prefixed with the component that they configure. For
|
||||
example, proper setting names for a fictional robots.txt extension would be
|
||||
``ROBOTSTXT_ENABLED``, ``ROBOTSTXT_OBEY``, ``ROBOTSTXT_CACHEDIR``, etc.
|
||||
The ``settings`` object can be used like a :class:`dict` (e.g.
|
||||
``settings["LOG_ENABLED"]``). However, to support non-string setting values,
|
||||
which may be passed from the command line as strings, it is recommended to use
|
||||
one of the methods provided by the :class:`~scrapy.settings.Settings` API.
|
||||
|
||||
|
||||
.. _component-priority-dictionaries:
|
||||
|
|
@ -1211,7 +1191,8 @@ EXTENSIONS
|
|||
|
||||
Default:: ``{}``
|
||||
|
||||
A dict containing the extensions enabled in your project, and their orders.
|
||||
:ref:`Component priority dictionary <component-priority-dictionaries>` of
|
||||
enabled extensions. See :ref:`topics-extensions`.
|
||||
|
||||
.. setting:: EXTENSIONS_BASE
|
||||
|
||||
|
|
|
|||
|
|
@ -63,17 +63,38 @@ particular setting. See each middleware documentation for more info.
|
|||
Writing your own spider middleware
|
||||
==================================
|
||||
|
||||
Each spider middleware is a Python class that defines one or more of the
|
||||
methods defined below.
|
||||
|
||||
The main entry point is the ``from_crawler`` class method, which receives a
|
||||
:class:`~scrapy.crawler.Crawler` instance. The :class:`~scrapy.crawler.Crawler`
|
||||
object gives you access, for example, to the :ref:`settings <topics-settings>`.
|
||||
Each spider middleware is a :ref:`component <topics-components>` that defines
|
||||
one or more of these methods:
|
||||
|
||||
.. module:: scrapy.spidermiddlewares
|
||||
|
||||
.. class:: SpiderMiddleware
|
||||
|
||||
.. method:: process_start_requests(start_requests, spider)
|
||||
|
||||
This method is called with the start requests of the spider, and works
|
||||
similarly to the :meth:`process_spider_output` method, except that it
|
||||
doesn't have a response associated and must return only requests (not
|
||||
items).
|
||||
|
||||
It receives an iterable (in the ``start_requests`` parameter) and must
|
||||
return another iterable of :class:`~scrapy.Request` objects and/or :ref:`item objects <topics-items>`.
|
||||
|
||||
.. note:: When implementing this method in your spider middleware, you
|
||||
should always return an iterable (that follows the input one) and
|
||||
not consume all ``start_requests`` iterator because it can be very
|
||||
large (or even unbounded) and cause a memory overflow. The Scrapy
|
||||
engine is designed to pull start requests while it has capacity to
|
||||
process them, so the start requests iterator can be effectively
|
||||
endless where there is some other condition for stopping the spider
|
||||
(like a time limit or item/page count).
|
||||
|
||||
:param start_requests: the start requests
|
||||
:type start_requests: an iterable of :class:`~scrapy.Request`
|
||||
|
||||
:param spider: the spider to whom the start requests belong
|
||||
:type spider: :class:`~scrapy.Spider` object
|
||||
|
||||
.. method:: process_spider_input(response, spider)
|
||||
|
||||
This method is called for each response that goes through the spider
|
||||
|
|
@ -168,42 +189,6 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
|
|||
:param spider: the spider which raised the exception
|
||||
:type spider: :class:`~scrapy.Spider` object
|
||||
|
||||
.. method:: process_start_requests(start_requests, spider)
|
||||
|
||||
This method is called with the start requests of the spider, and works
|
||||
similarly to the :meth:`process_spider_output` method, except that it
|
||||
doesn't have a response associated and must return only requests (not
|
||||
items).
|
||||
|
||||
It receives an iterable (in the ``start_requests`` parameter) and must
|
||||
return another iterable of :class:`~scrapy.Request` objects and/or :ref:`item objects <topics-items>`.
|
||||
|
||||
.. note:: When implementing this method in your spider middleware, you
|
||||
should always return an iterable (that follows the input one) and
|
||||
not consume all ``start_requests`` iterator because it can be very
|
||||
large (or even unbounded) and cause a memory overflow. The Scrapy
|
||||
engine is designed to pull start requests while it has capacity to
|
||||
process them, so the start requests iterator can be effectively
|
||||
endless where there is some other condition for stopping the spider
|
||||
(like a time limit or item/page count).
|
||||
|
||||
:param start_requests: the start requests
|
||||
:type start_requests: an iterable of :class:`~scrapy.Request`
|
||||
|
||||
:param spider: the spider to whom the start requests belong
|
||||
:type spider: :class:`~scrapy.Spider` object
|
||||
|
||||
.. method:: from_crawler(cls, crawler)
|
||||
|
||||
If present, this classmethod is called to create a middleware instance
|
||||
from a :class:`~scrapy.crawler.Crawler`. It must return a new instance
|
||||
of the middleware. Crawler object provides access to all Scrapy core
|
||||
components like settings and signals; it is a way for middleware to
|
||||
access them and hook its functionality into Scrapy.
|
||||
|
||||
:param crawler: crawler that uses this middleware
|
||||
:type crawler: :class:`~scrapy.crawler.Crawler` object
|
||||
|
||||
.. _topics-spider-middleware-ref:
|
||||
|
||||
Built-in spider middleware reference
|
||||
|
|
|
|||
Loading…
Reference in New Issue