mirror of https://github.com/scrapy/scrapy.git
--HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40701
This commit is contained in:
Pablo Hoffman
parent
300a0f4901
commit
bf0050c321
|
|
@ -0,0 +1,67 @@
|
|||
.. _ref-extension-manager:
|
||||
|
||||
=================
|
||||
Extension Manager
|
||||
=================
|
||||
|
||||
.. module:: scrapy.extension
|
||||
:synopsis: The extension manager
|
||||
|
||||
The Extension Manager is responsible for loading and keeping track of installed
|
||||
extensions and it's configured through the :setting:`EXTENSIONS` setting which
|
||||
contains a list of all available extensions.
|
||||
|
||||
The extension manager is a singleton-type class, which is instantiated at
|
||||
module load time and can be accessed like this::
|
||||
|
||||
from scrapy.extension import extensions
|
||||
webconsole_extension = extensions.enabled['WebConsole']
|
||||
|
||||
.. class:: ExtensionManager
|
||||
|
||||
Attributes
|
||||
----------
|
||||
|
||||
.. attribute:: ExtensionManager.loaded
|
||||
|
||||
A boolean which is True if extensions are already loaded or False if
|
||||
they're not.
|
||||
|
||||
.. attribute:: ExtensionManager.enabled
|
||||
|
||||
A dict with the enabled extensions. The keys are the extension class names,
|
||||
and the values are the extension objects. Example::
|
||||
|
||||
>>> from scrapy.extension import extensions
|
||||
>>> extensions.load()
|
||||
>>> print extensions.enabled
|
||||
{'CoreStats': <scrapy.stats.corestats.CoreStats object at 0x9e272ac>,
|
||||
'WebConsoke': <scrapy.management.telnet.TelnetConsole instance at 0xa05670c>,
|
||||
...
|
||||
|
||||
.. attribute:: ExtensionManager.disabled
|
||||
|
||||
A dict with the disabled extensions. The keys are the extension class names,
|
||||
and the values are the extension class paths (because objects are never
|
||||
instantiated for disabled extensions). Example::
|
||||
|
||||
>>> from scrapy.extension import extensions
|
||||
>>> extensions.load()
|
||||
>>> print extensions.disabled
|
||||
{'MemoryDebugger': 'scrapy.contrib.webconsole.stats.MemoryDebugger',
|
||||
'SpiderProfiler': 'scrapy.contrib.spider.profiler.SpiderProfiler',
|
||||
...
|
||||
|
||||
Methods
|
||||
-------
|
||||
|
||||
.. method:: ExtensionManager.load()
|
||||
|
||||
Load the available extensions configured in the :setting:`EXTENSIONS`
|
||||
setting. On a standard run, this method is usually called by the Execution
|
||||
Manager, but you may need to call it explicitly if you're dealing with
|
||||
code outside Scrapy.
|
||||
|
||||
.. method:: ExtensionManager.reload()
|
||||
|
||||
Reload the available extensions. See ``load()``.
|
||||
|
|
@ -0,0 +1,254 @@
|
|||
.. _ref-extensions:
|
||||
|
||||
=============================
|
||||
Built-in extensions reference
|
||||
=============================
|
||||
|
||||
This document explains all extensions that come with Scrapy. For information on
|
||||
how to use them and how to write your own extensions, see the :ref:`extensions
|
||||
usage guide <topics-extensions>`.
|
||||
|
||||
|
||||
General purpose extensions
|
||||
==========================
|
||||
|
||||
Core Stats extension
|
||||
--------------------
|
||||
|
||||
.. module:: scrapy.stats.corestats
|
||||
:synopsis: Core stats collection
|
||||
|
||||
.. class:: scrapy.stats.corestats.CoreStats
|
||||
|
||||
Enable the collection of core statistics, provided the stats collection are
|
||||
enabled (see :ref:`topics-stats`).
|
||||
|
||||
Response Libxml2 extension
|
||||
--------------------------
|
||||
|
||||
.. module:: scrapy.xpath.extension
|
||||
:synopsis: Libxml2 document caching for Responses
|
||||
|
||||
.. class:: scrapy.path.extension.ResponseLibxml2
|
||||
|
||||
Causes the :class:`~scrapy.http.Response` objects to grow a new method
|
||||
(``getlibxml2doc()``) which returns a (cached) libxml2 document of their
|
||||
contents. :ref:`XPath Selectors <topics-selectors>` use this extension for
|
||||
better performance, so it's highly recommended not to disable it.
|
||||
|
||||
.. _ref-extensions-webconsole:
|
||||
|
||||
Web console extension
|
||||
---------------------
|
||||
|
||||
.. module:: scrapy.management.web
|
||||
:synopsis: Web management console
|
||||
|
||||
.. class:: scrapy.management.web.WebConsole
|
||||
|
||||
Provides an extensible web server for managing a Scrapy process. It's enabled
|
||||
by the :setting:`WEBCONSOLE_ENABLED` setting. The server will listen in the
|
||||
port specified in :setting:`WEBCONSOLE_PORT`, and will log to the file
|
||||
specified in :setting:`WEBCONSOLE_LOGFILE`.
|
||||
|
||||
The web server is designed to be extended by other extensions which can add
|
||||
their own management web interfaces.
|
||||
|
||||
See also :ref:`topics-webconsole` for information on how to write your own web
|
||||
console extension, and "Web console extensions" below for a list of available
|
||||
built-in (web console) extensions.
|
||||
|
||||
Telnet console extension
|
||||
------------------------
|
||||
|
||||
.. module:: scrapy.management.telnet
|
||||
:synopsis: Telnet management console
|
||||
|
||||
.. class:: scrapy.management.telnet.TelnetConsole
|
||||
|
||||
Provides a telnet console for getting into a Python interpreter inside the
|
||||
currently running Scrapy process, which can be very useful for debugging.
|
||||
|
||||
The telnet console must be enabled by the :setting:`TELNETCONSOLE_ENABLED`
|
||||
setting, and the server will listen in the port specified in
|
||||
:setting:`WEBCONSOLE_PORT`.
|
||||
|
||||
Spider reloader extension
|
||||
-------------------------
|
||||
|
||||
.. module:: scrapy.contrib.spider.reloader
|
||||
:synopsis: Spider reloader extension
|
||||
|
||||
.. class:: scrapy.contrib.spider.reloader.SpiderReloader
|
||||
|
||||
Reload spider objects once they've finished scraping, to release the resources
|
||||
and references to other objects they may hold.
|
||||
|
||||
.. _ref-extensions-memusage:
|
||||
|
||||
Memory usage extension
|
||||
----------------------
|
||||
|
||||
.. module:: scrapy.contrib.memusage
|
||||
:synopsis: Memory usage extension
|
||||
|
||||
.. class:: scrapy.contrib.memusage.MemoryUsage
|
||||
|
||||
Allows monitoring the memory used by a Scrapy process and:
|
||||
|
||||
1, send a notification email when it exceeds a certain value
|
||||
2. terminate the Scrapy process when it exceeds a certain value
|
||||
|
||||
The notification emails can be triggered when a certain warning value is
|
||||
reached (:setting:`MEMUSAGE_WARNING_MB`) and when the maximum value is reached
|
||||
(:setting:`MEMUSAGE_LIMIT_MB`) which will also cause the Scrapy process to be
|
||||
terminated.
|
||||
|
||||
This extension is enabled by the :setting:`MEMUSAGE_ENABLED` setting and
|
||||
can be configured with the following settings:
|
||||
|
||||
* :setting:`MEMUSAGE_LIMIT_MB`
|
||||
* :setting:`MEMUSAGE_WARNING_MB`
|
||||
* :setting:`MEMUSAGE_NOTIFY_MAIL`
|
||||
* :setting:`MEMUSAGE_REPORT`
|
||||
|
||||
Memory debugger extension
|
||||
-------------------------
|
||||
|
||||
.. module:: scrapy.contrib.memdebug
|
||||
:synopsis: Memory debugger extension
|
||||
|
||||
.. class:: scrapy.contrib.memdebug.MemoryDebugger
|
||||
|
||||
A memory debugger which collects some info about objects uncollected by the
|
||||
garbage collector and libxml2 memory leaks. To enable this extension turn on
|
||||
the :setting:`MEMDEBUG_ENABLED` setting. The report will be printed to standard
|
||||
output. If the :setting:`MEMDEBUG_NOTIFY` setting contains a list of emails the
|
||||
report will also be sent to those addresses.
|
||||
|
||||
Close domain extension
|
||||
----------------------
|
||||
|
||||
.. module:: scrapy.contrib.closedomain
|
||||
:synopsis: Close domain extension
|
||||
|
||||
.. class:: scrapy.contrib.closedomain.CloseDomain
|
||||
|
||||
Allows a domain/spider to be automatically closed when some conditions are met
|
||||
and optionally notifies a list of emails if the :setting:`CLOSEDOMAIN_NOTIFY`
|
||||
is set.
|
||||
|
||||
The conditions for auto-closing a domain can be configured thorough these
|
||||
settings:
|
||||
|
||||
* :setting:`CLOSEDOMAIN_TIMEOUT` - an integer which specifies a number of
|
||||
seconds. If the domain remains open for more than that time, it will be
|
||||
closed.
|
||||
|
||||
Other conditions will be supported in the future.
|
||||
|
||||
Stack trace debug extension
|
||||
---------------------------
|
||||
|
||||
.. module:: scrapy.contrib.debug
|
||||
:synopsis: Extensions for debugging Scrapy
|
||||
|
||||
.. class:: scrapy.contrib.debug.StackTraceDebug
|
||||
|
||||
Adds a `SIGUSR1`_ signal handler which dumps the stack trace of a runnning
|
||||
Scrapy process when a ``SIGUSR1`` signal is catched. After the stack trace is
|
||||
dumped, the Scrapy process continues to run normally.
|
||||
|
||||
The stack trace is sent to standard output, or to the Scrapy log file if
|
||||
:setting:`LOG_STDOUT` is enabled.
|
||||
|
||||
This extension only works on POSIX-compliant platforms (ie. not Windows).
|
||||
|
||||
.. _SIGUSR1: http://en.wikipedia.org/wiki/SIGUSR1_and_SIGUSR2
|
||||
|
||||
Response soup extension
|
||||
-----------------------
|
||||
|
||||
.. module:: scrapy.contrib.response.soup
|
||||
:synopsis: Response soup extension
|
||||
|
||||
.. class:: scrapy.contrib.response.soup.ResponseSoup
|
||||
|
||||
The ResponseSoup extension causes the :class:`~scrapy.http.Response` objects to
|
||||
grow a new method (``getsoup()``) which returns a cached `BeautifulSoup`_
|
||||
object of their body, and a ``soup`` attribute with the same effect. The
|
||||
``soup`` attribute is provided only for convenience, as you cannot pass pass
|
||||
any BeautifulSoup constructor arguments (use the ``getsoup()`` method for those
|
||||
cases).
|
||||
|
||||
The advantage of using the Response soup extension over instantiating a
|
||||
BeautifulSoup object directly is performance, as BeautifulSoup is known to be
|
||||
very slow.
|
||||
|
||||
For example, if you have a downloader middleware and a spider that both need to
|
||||
construct a BeautifulSoup object of the responses, you would be constructing
|
||||
two BeautifulSoup objects unless you use this extension which caches the first
|
||||
one.
|
||||
|
||||
.. _BeautifulSoup: http://www.crummy.com/software/BeautifulSoup/documentation.html
|
||||
|
||||
|
||||
Web console extensions
|
||||
======================
|
||||
|
||||
.. module:: scrapy.contrib.webconsole
|
||||
:synopsis: Contains most built-in web console extensions
|
||||
|
||||
Here is a list of built-in web console extensions. For clarity "web console
|
||||
extension" is abbreviated as "WC extension".
|
||||
|
||||
For more information see the see the :ref:`web console documentation
|
||||
<topics-webconsole>`.
|
||||
|
||||
Spider live stats WC extension
|
||||
------------------------------
|
||||
|
||||
.. module:: scrapy.contrib.webconsole.livestats
|
||||
:synopsis: Spider live stats web console extension
|
||||
|
||||
.. class:: scrapy.contrib.webconsole.livestats.LiveStats
|
||||
|
||||
Display a table with stats of all spider crawled by the current Scrapy run,
|
||||
including:
|
||||
|
||||
* Number of items scraped
|
||||
* Number of pages crawled
|
||||
* Number of pending requests in the scheduler
|
||||
* Number of pending requests in the downloader queue
|
||||
* Number of requests currently being downloaded
|
||||
|
||||
Engine status WC extension
|
||||
---------------------------
|
||||
|
||||
.. module:: scrapy.contrib.webconsole.enginestatus
|
||||
:synopsis: Engine status web console extension
|
||||
|
||||
.. class:: scrapy.contrib.webconsole.enginestatus.EngineStatus
|
||||
|
||||
Display the current status of the Scrapy Engine, which is just the output of
|
||||
the Scrapy engine ``getstatus()`` method.
|
||||
|
||||
Stats collector dump WC extension
|
||||
----------------------------------
|
||||
|
||||
.. module:: scrapy.contrib.webconsole.stats
|
||||
:synopsis: Stats dump web console extension
|
||||
|
||||
.. class:: scrapy.contrib.webconsole.stats.StatsDump
|
||||
|
||||
Display the stats collected so far by the stats collector.
|
||||
|
||||
Spider stats WC extension
|
||||
-------------------------
|
||||
|
||||
.. module:: scrapy.contrib.webconsole.spiderstats
|
||||
:synopsis: Spider stats web console extension
|
||||
|
||||
.. class:: scrapy.contrib.webconsole.spiderstats.SpiderStats
|
||||
|
||||
|
||||
|
|
@ -10,5 +10,7 @@ This section documents the API of Scrapy |version|. For more information see :re
|
|||
|
||||
exceptions
|
||||
request-response
|
||||
extension-manager
|
||||
extensions
|
||||
settings
|
||||
signals
|
||||
|
|
|
|||
|
|
@ -77,7 +77,7 @@ CACHE2_SECTORIZE
|
|||
|
||||
Default: ``True``
|
||||
|
||||
Wether to split HTTP cache storage in several dirs for performance improvements.
|
||||
Whether to split HTTP cache storage in several dirs for performance improvements.
|
||||
|
||||
.. setting:: CLOSEDOMAIN_NOTIFY
|
||||
|
||||
|
|
@ -269,7 +269,7 @@ DEPTH_STATS
|
|||
|
||||
Default: ``True``
|
||||
|
||||
Wether to collect depth stats.
|
||||
Whether to collect depth stats.
|
||||
|
||||
.. setting:: DOWNLOADER_MIDDLEWARES
|
||||
|
||||
|
|
@ -304,7 +304,7 @@ DOWNLOADER_STATS
|
|||
|
||||
Default: ``True``
|
||||
|
||||
Wether to enable downloader stats collection.
|
||||
Whether to enable downloader stats collection.
|
||||
|
||||
.. setting:: DOWNLOAD_TIMEOUT
|
||||
|
||||
|
|
@ -338,7 +338,6 @@ Default::
|
|||
'scrapy.xpath.extension.ResponseLibxml2',
|
||||
'scrapy.management.web.WebConsole',
|
||||
'scrapy.management.telnet.TelnetConsole',
|
||||
'scrapy.contrib.webconsole.schedstats.SchedulerStats',
|
||||
'scrapy.contrib.webconsole.livestats.LiveStats',
|
||||
'scrapy.contrib.webconsole.spiderctl.Spiderctl',
|
||||
'scrapy.contrib.webconsole.enginestatus.EngineStatus',
|
||||
|
|
@ -355,8 +354,12 @@ Default::
|
|||
'scrapy.contrib.response.soup.ResponseSoup',
|
||||
]
|
||||
|
||||
The list of enabled extensions. Keep in mind that some of them may also need to
|
||||
be activated through a setting.
|
||||
The list of available extensions. Keep in mind that some of them need need to
|
||||
be enabled through a setting. By default, this setting contains all stable
|
||||
built-in extensions.
|
||||
|
||||
For more information See the :ref:`extensions user guide <topics-extensions>`
|
||||
and the :ref:`list of available extensions <ref-extensions>`.
|
||||
|
||||
.. setting:: GROUPSETTINGS_ENABLED
|
||||
|
||||
|
|
@ -365,7 +368,7 @@ GROUPSETTINGS_ENABLED
|
|||
|
||||
Default: ``False``
|
||||
|
||||
Wether to enable group settings where spiders pull their settings from.
|
||||
Whether to enable group settings where spiders pull their settings from.
|
||||
|
||||
.. setting:: GROUPSETTINGS_MODULE
|
||||
|
||||
|
|
@ -449,14 +452,18 @@ MEMDEBUG_ENABLED
|
|||
|
||||
Default: ``False``
|
||||
|
||||
Wether to enable memory debugging.
|
||||
Whether to enable memory debugging.
|
||||
|
||||
.. setting:: MEMDEBUG_NOTIFY
|
||||
|
||||
MEMDEBUG_NOTIFY
|
||||
---------------
|
||||
|
||||
Default: ``[]``
|
||||
|
||||
If memory debugging is enabled a memory report will be sent to the specified
|
||||
addresses.
|
||||
When memory debugging is enabled a memory report will be sent to the specified
|
||||
addresses if this setting is not empty, otherwise the report will be written to
|
||||
the log.
|
||||
|
||||
Example::
|
||||
|
||||
|
|
@ -471,10 +478,12 @@ Default: ``False``
|
|||
|
||||
Scope: ``scrapy.contrib.memusage``
|
||||
|
||||
Wether to enable the memory usage extension that will shutdown the Scrapy
|
||||
Whether to enable the memory usage extension that will shutdown the Scrapy
|
||||
process when it exceeds a memory limit, and also notify by email when that
|
||||
happened.
|
||||
|
||||
See :ref:`ref-extensions-memusage`.
|
||||
|
||||
.. setting:: MEMUSAGE_LIMIT_MB
|
||||
|
||||
MEMUSAGE_LIMIT_MB
|
||||
|
|
@ -487,6 +496,8 @@ Scope: ``scrapy.contrib.memusage``
|
|||
The maximum amount of memory to allow (in megabytes) before shutting down
|
||||
Scrapy (if MEMUSAGE_ENABLED is True). If zero, no check will be performed.
|
||||
|
||||
See :ref:`ref-extensions-memusage`.
|
||||
|
||||
.. setting:: MEMUSAGE_NOTIFY_MAIL
|
||||
|
||||
MEMUSAGE_NOTIFY_MAIL
|
||||
|
|
@ -502,6 +513,8 @@ Example::
|
|||
|
||||
MEMUSAGE_NOTIFY_MAIL = ['user@example.com']
|
||||
|
||||
See :ref:`ref-extensions-memusage`.
|
||||
|
||||
.. setting:: MEMUSAGE_REPORT
|
||||
|
||||
MEMUSAGE_REPORT
|
||||
|
|
@ -511,12 +524,14 @@ Default: ``False``
|
|||
|
||||
Scope: ``scrapy.contrib.memusage``
|
||||
|
||||
Wether to send a memory usage report after each domain has been closed.
|
||||
Whether to send a memory usage report after each domain has been closed.
|
||||
|
||||
See :ref:`ref-extensions-memusage`.
|
||||
|
||||
.. setting:: MEMUSAGE_WARNING_MB
|
||||
|
||||
MEMUSAGE_LIMIT_MB
|
||||
-----------------
|
||||
MEMUSAGE_WARNING_MB
|
||||
-------------------
|
||||
|
||||
Default: ``0``
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,103 @@
|
|||
.. _topics-extensions:
|
||||
|
||||
==========
|
||||
Extensions
|
||||
==========
|
||||
|
||||
The extensions framework provide a mechanism for inserting your own
|
||||
custom functionality into Scrapy.
|
||||
|
||||
Extensions are just regular classes that are instantiated at Scrapy startup,
|
||||
when extensions are initialized.
|
||||
|
||||
Extension settings
|
||||
==================
|
||||
|
||||
Extensions use the :ref:`Scrapy settings <topics-settings>` to manage their
|
||||
settings, just like any other Scrapy code.
|
||||
|
||||
It is customary for extensions to prefix their settings with their own name, to
|
||||
avoid collision with existing (and future) extensions. For example, an
|
||||
hypothetic extension to handle `Google Sitemaps`_ would use settings like
|
||||
`GOOGLESITEMAP_ENABLED`, `GOOGLESITEMAP_DEPTH`, and so on.
|
||||
|
||||
.. _Google Sitemaps: http://en.wikipedia.org/wiki/Sitemaps
|
||||
|
||||
Loading & activating extensions
|
||||
===============================
|
||||
|
||||
Extensions are loaded and activated at startup by instantiating a single
|
||||
instance of the extension class. Therefore, all the extension initialization
|
||||
code must be performed in the class constructor (``__init__`` method).
|
||||
|
||||
To make an extension available, add it to the :setting:`EXTENSIONS` list in
|
||||
your Scrapy settings. In :setting:`EXTENSIONS`, each extension is represented
|
||||
by a string: the full Python path to the extension's class name. For example::
|
||||
|
||||
EXTENSIONS = [
|
||||
'scrapy.stats.corestats.CoreStats',
|
||||
'scrapy.management.web.WebConsole',
|
||||
'scrapy.management.telnet.TelnetConsole',
|
||||
'scrapy.contrib.webconsole.enginestatus.EngineStatus',
|
||||
'scrapy.contrib.webconsole.stats.StatsDump',
|
||||
'scrapy.contrib.debug.StackTraceDebug',
|
||||
]
|
||||
|
||||
|
||||
Available, enabled and disabled extensions
|
||||
==========================================
|
||||
|
||||
Not all available extensions will be enabled. Some of them usually depend on a
|
||||
particular setting. For example, the Cache extension is typically available but
|
||||
disabled by default unless you the :setting:`CACHE2_DIR` setting is set. Both
|
||||
enabled and disabled extension can be accessed through the
|
||||
:ref:`ref-extension-manager`.
|
||||
|
||||
Accessing enabled extensions
|
||||
============================
|
||||
|
||||
Even though it's not usually needed, you can access extension objects through
|
||||
the :ref:`ref-extension-manager` which is populated when extensions are loaded.
|
||||
For example, to access the ``WebConsole`` extension::
|
||||
|
||||
from scrapy.extension import extensions
|
||||
webconsole_extension = extensions.enabled['WebConsole']
|
||||
|
||||
.. seealso::
|
||||
|
||||
:ref:`ref-extension-manager`, for the complete Extension manager reference.
|
||||
|
||||
Writing your own extension
|
||||
==========================
|
||||
|
||||
Writing your own extension is easy. Each extension is a single Python class
|
||||
which doesn't need to implement any particular method.
|
||||
|
||||
All extension initialization code must be performed in the class constructor
|
||||
(``__init__`` method). If that method raises the :exception:`NotConfigured`
|
||||
exception, the extension will be disabled. Otherwise, the extension will be
|
||||
enabled.
|
||||
|
||||
Let's take a look at the following example extension which just logs a message
|
||||
everytime a domain/spider is opened and closed::
|
||||
|
||||
from pydispatch import dispatcher
|
||||
from scrapy.core import signals
|
||||
|
||||
class SpiderOpenCloseLogging(object):
|
||||
|
||||
def __init__(self):
|
||||
dispatcher.connect(self.domain_opened, signal=signals.domain_opened)
|
||||
dispatcher.connect(self.domain_closed, signal=signals.domain_closed)
|
||||
|
||||
def domain_opened(self, domain, spider):
|
||||
log.msg("opened domain %s" % domain)
|
||||
|
||||
def domain_closed(self, domain, spider):
|
||||
log.msg("closed domain %s" % domain)
|
||||
|
||||
Built-in extensions
|
||||
===================
|
||||
|
||||
See :ref:`ref-extensions`.
|
||||
|
||||
|
|
@ -17,4 +17,7 @@ This section introduces all key concepts of Scrapy.
|
|||
downloader-middleware
|
||||
spider-middleware
|
||||
settings
|
||||
extensions
|
||||
stats
|
||||
webconsole
|
||||
robotstxt
|
||||
|
|
|
|||
|
|
@ -0,0 +1,8 @@
|
|||
.. _topics-stats:
|
||||
|
||||
Stats
|
||||
=====
|
||||
|
||||
Scrapy provides built-in functionality for collecting stats in a tree-like
|
||||
structure.
|
||||
|
||||
|
|
@ -0,0 +1,83 @@
|
|||
.. _topics-webconsole:
|
||||
|
||||
===========
|
||||
Web Console
|
||||
===========
|
||||
|
||||
Scrapy comes with a built-in web server for monitoring and controlling a Scrapy
|
||||
running process.
|
||||
|
||||
The web console is :ref:`built-in Scrapy extension <ref-extensions>` which
|
||||
comes enabled by default, but you can also disable it if you're running tight
|
||||
on memory.
|
||||
|
||||
For more information about this extension see :ref:`ref-extensions-webconsole`.
|
||||
|
||||
Writing a web console extension
|
||||
===============================
|
||||
|
||||
Writing a web console extension is similar to writing any other :ref:`Scrapy
|
||||
extensions <topics-extensions>` except that the extension class must:
|
||||
|
||||
1. catch the ``scrapy.management.web.webconsole_discover_module`` signal, and
|
||||
return itself in the handler.
|
||||
|
||||
2. have the following two attributes:
|
||||
|
||||
.. attribute:: webconsole_id
|
||||
|
||||
The id by which the Scrapy web interface will known this extension, and
|
||||
also the main dir under which this extension interface will work. For
|
||||
example, assuming Scrapy web server is listening on
|
||||
http://localhost:8000/ and the ``webconsole_id='extension1'`` the web
|
||||
main page for the interface of that extension will be:
|
||||
|
||||
http://localhost:8000/extension1/
|
||||
|
||||
.. attribute:: webconsole_name
|
||||
|
||||
The name by which the Scrapy web server will know that extension. That name
|
||||
will be displayed in the main web console index, as the text that links to
|
||||
the extension main page.
|
||||
|
||||
3. implement the following method:
|
||||
|
||||
.. method:: webconsole_render(wc_request)
|
||||
|
||||
``wc_request`` is a `twisted.web.http.Request`_ object with the HTTP request
|
||||
sent to the web console.
|
||||
|
||||
.. _twisted.web.http.Request: http://python.net/crew/mwh/apidocs/twisted.web.http.Request.html
|
||||
|
||||
It must return a str with the web page to render, typically containing HTML
|
||||
code.
|
||||
|
||||
Example web console extension
|
||||
=============================
|
||||
|
||||
Here's an example of a simple web console extension that just displays a "Hello
|
||||
world!" text::
|
||||
|
||||
from pydispatch import dispatcher
|
||||
from scrapy.management.web import webconsole_discover_module
|
||||
|
||||
class HelloWorldConsole(object):
|
||||
webconsole_id = 'helloworld'
|
||||
webconsole_name = 'Hello world'
|
||||
|
||||
def __init__(self):
|
||||
dispatcher.connect(self.webconsole_discover_module, signal=webconsole_discover_module)
|
||||
|
||||
def webconsole_discover_module(self):
|
||||
return self
|
||||
|
||||
def webconsole_render(self, wc_request):
|
||||
return "<html><head></head><body><h1>Hello world!</h1></body>"
|
||||
|
||||
If you start Scrapy with the web console enabled on http://localhost:8000/ and
|
||||
you access the URL:
|
||||
|
||||
http://localhost:8000/helloworld/
|
||||
|
||||
You will see a page containing a big "Hello World!" text.
|
||||
|
||||
Loading…
Reference in New Issue