diff --git a/scrapy/trunk/docs/ref/extension-manager.rst b/scrapy/trunk/docs/ref/extension-manager.rst new file mode 100644 index 000000000..96fe0a188 --- /dev/null +++ b/scrapy/trunk/docs/ref/extension-manager.rst @@ -0,0 +1,67 @@ +.. _ref-extension-manager: + +================= +Extension Manager +================= + +.. module:: scrapy.extension + :synopsis: The extension manager + +The Extension Manager is responsible for loading and keeping track of installed +extensions and it's configured through the :setting:`EXTENSIONS` setting which +contains a list of all available extensions. + +The extension manager is a singleton-type class, which is instantiated at +module load time and can be accessed like this:: + + from scrapy.extension import extensions + webconsole_extension = extensions.enabled['WebConsole'] + +.. class:: ExtensionManager + +Attributes +---------- + +.. attribute:: ExtensionManager.loaded + + A boolean which is True if extensions are already loaded or False if + they're not. + +.. attribute:: ExtensionManager.enabled + + A dict with the enabled extensions. The keys are the extension class names, + and the values are the extension objects. Example:: + + >>> from scrapy.extension import extensions + >>> extensions.load() + >>> print extensions.enabled + {'CoreStats': , + 'WebConsoke': , + ... + +.. attribute:: ExtensionManager.disabled + + A dict with the disabled extensions. The keys are the extension class names, + and the values are the extension class paths (because objects are never + instantiated for disabled extensions). Example:: + + >>> from scrapy.extension import extensions + >>> extensions.load() + >>> print extensions.disabled + {'MemoryDebugger': 'scrapy.contrib.webconsole.stats.MemoryDebugger', + 'SpiderProfiler': 'scrapy.contrib.spider.profiler.SpiderProfiler', + ... + +Methods +------- + +.. method:: ExtensionManager.load() + + Load the available extensions configured in the :setting:`EXTENSIONS` + setting. On a standard run, this method is usually called by the Execution + Manager, but you may need to call it explicitly if you're dealing with + code outside Scrapy. + +.. method:: ExtensionManager.reload() + + Reload the available extensions. See ``load()``. diff --git a/scrapy/trunk/docs/ref/extensions.rst b/scrapy/trunk/docs/ref/extensions.rst new file mode 100644 index 000000000..9a0d26992 --- /dev/null +++ b/scrapy/trunk/docs/ref/extensions.rst @@ -0,0 +1,254 @@ +.. _ref-extensions: + +============================= +Built-in extensions reference +============================= + +This document explains all extensions that come with Scrapy. For information on +how to use them and how to write your own extensions, see the :ref:`extensions +usage guide `. + + +General purpose extensions +========================== + +Core Stats extension +-------------------- + +.. module:: scrapy.stats.corestats + :synopsis: Core stats collection + +.. class:: scrapy.stats.corestats.CoreStats + +Enable the collection of core statistics, provided the stats collection are +enabled (see :ref:`topics-stats`). + +Response Libxml2 extension +-------------------------- + +.. module:: scrapy.xpath.extension + :synopsis: Libxml2 document caching for Responses + +.. class:: scrapy.path.extension.ResponseLibxml2 + +Causes the :class:`~scrapy.http.Response` objects to grow a new method +(``getlibxml2doc()``) which returns a (cached) libxml2 document of their +contents. :ref:`XPath Selectors ` use this extension for +better performance, so it's highly recommended not to disable it. + +.. _ref-extensions-webconsole: + +Web console extension +--------------------- + +.. module:: scrapy.management.web + :synopsis: Web management console + +.. class:: scrapy.management.web.WebConsole + +Provides an extensible web server for managing a Scrapy process. It's enabled +by the :setting:`WEBCONSOLE_ENABLED` setting. The server will listen in the +port specified in :setting:`WEBCONSOLE_PORT`, and will log to the file +specified in :setting:`WEBCONSOLE_LOGFILE`. + +The web server is designed to be extended by other extensions which can add +their own management web interfaces. + +See also :ref:`topics-webconsole` for information on how to write your own web +console extension, and "Web console extensions" below for a list of available +built-in (web console) extensions. + +Telnet console extension +------------------------ + +.. module:: scrapy.management.telnet + :synopsis: Telnet management console + +.. class:: scrapy.management.telnet.TelnetConsole + +Provides a telnet console for getting into a Python interpreter inside the +currently running Scrapy process, which can be very useful for debugging. + +The telnet console must be enabled by the :setting:`TELNETCONSOLE_ENABLED` +setting, and the server will listen in the port specified in +:setting:`WEBCONSOLE_PORT`. + +Spider reloader extension +------------------------- + +.. module:: scrapy.contrib.spider.reloader + :synopsis: Spider reloader extension + +.. class:: scrapy.contrib.spider.reloader.SpiderReloader + +Reload spider objects once they've finished scraping, to release the resources +and references to other objects they may hold. + +.. _ref-extensions-memusage: + +Memory usage extension +---------------------- + +.. module:: scrapy.contrib.memusage + :synopsis: Memory usage extension + +.. class:: scrapy.contrib.memusage.MemoryUsage + +Allows monitoring the memory used by a Scrapy process and: + +1, send a notification email when it exceeds a certain value +2. terminate the Scrapy process when it exceeds a certain value + +The notification emails can be triggered when a certain warning value is +reached (:setting:`MEMUSAGE_WARNING_MB`) and when the maximum value is reached +(:setting:`MEMUSAGE_LIMIT_MB`) which will also cause the Scrapy process to be +terminated. + +This extension is enabled by the :setting:`MEMUSAGE_ENABLED` setting and +can be configured with the following settings: + +* :setting:`MEMUSAGE_LIMIT_MB` +* :setting:`MEMUSAGE_WARNING_MB` +* :setting:`MEMUSAGE_NOTIFY_MAIL` +* :setting:`MEMUSAGE_REPORT` + +Memory debugger extension +------------------------- + +.. module:: scrapy.contrib.memdebug + :synopsis: Memory debugger extension + +.. class:: scrapy.contrib.memdebug.MemoryDebugger + +A memory debugger which collects some info about objects uncollected by the +garbage collector and libxml2 memory leaks. To enable this extension turn on +the :setting:`MEMDEBUG_ENABLED` setting. The report will be printed to standard +output. If the :setting:`MEMDEBUG_NOTIFY` setting contains a list of emails the +report will also be sent to those addresses. + +Close domain extension +---------------------- + +.. module:: scrapy.contrib.closedomain + :synopsis: Close domain extension + +.. class:: scrapy.contrib.closedomain.CloseDomain + +Allows a domain/spider to be automatically closed when some conditions are met +and optionally notifies a list of emails if the :setting:`CLOSEDOMAIN_NOTIFY` +is set. + +The conditions for auto-closing a domain can be configured thorough these +settings: + +* :setting:`CLOSEDOMAIN_TIMEOUT` - an integer which specifies a number of + seconds. If the domain remains open for more than that time, it will be + closed. + +Other conditions will be supported in the future. + +Stack trace debug extension +--------------------------- + +.. module:: scrapy.contrib.debug + :synopsis: Extensions for debugging Scrapy + +.. class:: scrapy.contrib.debug.StackTraceDebug + +Adds a `SIGUSR1`_ signal handler which dumps the stack trace of a runnning +Scrapy process when a ``SIGUSR1`` signal is catched. After the stack trace is +dumped, the Scrapy process continues to run normally. + +The stack trace is sent to standard output, or to the Scrapy log file if +:setting:`LOG_STDOUT` is enabled. + +This extension only works on POSIX-compliant platforms (ie. not Windows). + +.. _SIGUSR1: http://en.wikipedia.org/wiki/SIGUSR1_and_SIGUSR2 + +Response soup extension +----------------------- + +.. module:: scrapy.contrib.response.soup + :synopsis: Response soup extension + +.. class:: scrapy.contrib.response.soup.ResponseSoup + +The ResponseSoup extension causes the :class:`~scrapy.http.Response` objects to +grow a new method (``getsoup()``) which returns a cached `BeautifulSoup`_ +object of their body, and a ``soup`` attribute with the same effect. The +``soup`` attribute is provided only for convenience, as you cannot pass pass +any BeautifulSoup constructor arguments (use the ``getsoup()`` method for those +cases). + +The advantage of using the Response soup extension over instantiating a +BeautifulSoup object directly is performance, as BeautifulSoup is known to be +very slow. + +For example, if you have a downloader middleware and a spider that both need to +construct a BeautifulSoup object of the responses, you would be constructing +two BeautifulSoup objects unless you use this extension which caches the first +one. + +.. _BeautifulSoup: http://www.crummy.com/software/BeautifulSoup/documentation.html + + +Web console extensions +====================== + +.. module:: scrapy.contrib.webconsole + :synopsis: Contains most built-in web console extensions + +Here is a list of built-in web console extensions. For clarity "web console +extension" is abbreviated as "WC extension". + +For more information see the see the :ref:`web console documentation +`. + +Spider live stats WC extension +------------------------------ + +.. module:: scrapy.contrib.webconsole.livestats + :synopsis: Spider live stats web console extension + +.. class:: scrapy.contrib.webconsole.livestats.LiveStats + +Display a table with stats of all spider crawled by the current Scrapy run, +including: + +* Number of items scraped +* Number of pages crawled +* Number of pending requests in the scheduler +* Number of pending requests in the downloader queue +* Number of requests currently being downloaded + +Engine status WC extension +--------------------------- + +.. module:: scrapy.contrib.webconsole.enginestatus + :synopsis: Engine status web console extension + +.. class:: scrapy.contrib.webconsole.enginestatus.EngineStatus + +Display the current status of the Scrapy Engine, which is just the output of +the Scrapy engine ``getstatus()`` method. + +Stats collector dump WC extension +---------------------------------- + +.. module:: scrapy.contrib.webconsole.stats + :synopsis: Stats dump web console extension + +.. class:: scrapy.contrib.webconsole.stats.StatsDump + +Display the stats collected so far by the stats collector. + +Spider stats WC extension +------------------------- + +.. module:: scrapy.contrib.webconsole.spiderstats + :synopsis: Spider stats web console extension + +.. class:: scrapy.contrib.webconsole.spiderstats.SpiderStats + + diff --git a/scrapy/trunk/docs/ref/index.rst b/scrapy/trunk/docs/ref/index.rst index d61f35b12..98d2fe0c5 100644 --- a/scrapy/trunk/docs/ref/index.rst +++ b/scrapy/trunk/docs/ref/index.rst @@ -10,5 +10,7 @@ This section documents the API of Scrapy |version|. For more information see :re exceptions request-response + extension-manager + extensions settings signals diff --git a/scrapy/trunk/docs/ref/settings.rst b/scrapy/trunk/docs/ref/settings.rst index a364f4201..125b3e16e 100644 --- a/scrapy/trunk/docs/ref/settings.rst +++ b/scrapy/trunk/docs/ref/settings.rst @@ -77,7 +77,7 @@ CACHE2_SECTORIZE Default: ``True`` -Wether to split HTTP cache storage in several dirs for performance improvements. +Whether to split HTTP cache storage in several dirs for performance improvements. .. setting:: CLOSEDOMAIN_NOTIFY @@ -269,7 +269,7 @@ DEPTH_STATS Default: ``True`` -Wether to collect depth stats. +Whether to collect depth stats. .. setting:: DOWNLOADER_MIDDLEWARES @@ -304,7 +304,7 @@ DOWNLOADER_STATS Default: ``True`` -Wether to enable downloader stats collection. +Whether to enable downloader stats collection. .. setting:: DOWNLOAD_TIMEOUT @@ -338,7 +338,6 @@ Default:: 'scrapy.xpath.extension.ResponseLibxml2', 'scrapy.management.web.WebConsole', 'scrapy.management.telnet.TelnetConsole', - 'scrapy.contrib.webconsole.schedstats.SchedulerStats', 'scrapy.contrib.webconsole.livestats.LiveStats', 'scrapy.contrib.webconsole.spiderctl.Spiderctl', 'scrapy.contrib.webconsole.enginestatus.EngineStatus', @@ -355,8 +354,12 @@ Default:: 'scrapy.contrib.response.soup.ResponseSoup', ] -The list of enabled extensions. Keep in mind that some of them may also need to -be activated through a setting. +The list of available extensions. Keep in mind that some of them need need to +be enabled through a setting. By default, this setting contains all stable +built-in extensions. + +For more information See the :ref:`extensions user guide ` +and the :ref:`list of available extensions `. .. setting:: GROUPSETTINGS_ENABLED @@ -365,7 +368,7 @@ GROUPSETTINGS_ENABLED Default: ``False`` -Wether to enable group settings where spiders pull their settings from. +Whether to enable group settings where spiders pull their settings from. .. setting:: GROUPSETTINGS_MODULE @@ -449,14 +452,18 @@ MEMDEBUG_ENABLED Default: ``False`` -Wether to enable memory debugging. +Whether to enable memory debugging. .. setting:: MEMDEBUG_NOTIFY +MEMDEBUG_NOTIFY +--------------- + Default: ``[]`` -If memory debugging is enabled a memory report will be sent to the specified -addresses. +When memory debugging is enabled a memory report will be sent to the specified +addresses if this setting is not empty, otherwise the report will be written to +the log. Example:: @@ -471,10 +478,12 @@ Default: ``False`` Scope: ``scrapy.contrib.memusage`` -Wether to enable the memory usage extension that will shutdown the Scrapy +Whether to enable the memory usage extension that will shutdown the Scrapy process when it exceeds a memory limit, and also notify by email when that happened. +See :ref:`ref-extensions-memusage`. + .. setting:: MEMUSAGE_LIMIT_MB MEMUSAGE_LIMIT_MB @@ -487,6 +496,8 @@ Scope: ``scrapy.contrib.memusage`` The maximum amount of memory to allow (in megabytes) before shutting down Scrapy (if MEMUSAGE_ENABLED is True). If zero, no check will be performed. +See :ref:`ref-extensions-memusage`. + .. setting:: MEMUSAGE_NOTIFY_MAIL MEMUSAGE_NOTIFY_MAIL @@ -502,6 +513,8 @@ Example:: MEMUSAGE_NOTIFY_MAIL = ['user@example.com'] +See :ref:`ref-extensions-memusage`. + .. setting:: MEMUSAGE_REPORT MEMUSAGE_REPORT @@ -511,12 +524,14 @@ Default: ``False`` Scope: ``scrapy.contrib.memusage`` -Wether to send a memory usage report after each domain has been closed. +Whether to send a memory usage report after each domain has been closed. + +See :ref:`ref-extensions-memusage`. .. setting:: MEMUSAGE_WARNING_MB -MEMUSAGE_LIMIT_MB ------------------ +MEMUSAGE_WARNING_MB +------------------- Default: ``0`` diff --git a/scrapy/trunk/docs/topics/extensions.rst b/scrapy/trunk/docs/topics/extensions.rst new file mode 100644 index 000000000..73703ce2f --- /dev/null +++ b/scrapy/trunk/docs/topics/extensions.rst @@ -0,0 +1,103 @@ +.. _topics-extensions: + +========== +Extensions +========== + +The extensions framework provide a mechanism for inserting your own +custom functionality into Scrapy. + +Extensions are just regular classes that are instantiated at Scrapy startup, +when extensions are initialized. + +Extension settings +================== + +Extensions use the :ref:`Scrapy settings ` to manage their +settings, just like any other Scrapy code. + +It is customary for extensions to prefix their settings with their own name, to +avoid collision with existing (and future) extensions. For example, an +hypothetic extension to handle `Google Sitemaps`_ would use settings like +`GOOGLESITEMAP_ENABLED`, `GOOGLESITEMAP_DEPTH`, and so on. + +.. _Google Sitemaps: http://en.wikipedia.org/wiki/Sitemaps + +Loading & activating extensions +=============================== + +Extensions are loaded and activated at startup by instantiating a single +instance of the extension class. Therefore, all the extension initialization +code must be performed in the class constructor (``__init__`` method). + +To make an extension available, add it to the :setting:`EXTENSIONS` list in +your Scrapy settings. In :setting:`EXTENSIONS`, each extension is represented +by a string: the full Python path to the extension's class name. For example:: + + EXTENSIONS = [ + 'scrapy.stats.corestats.CoreStats', + 'scrapy.management.web.WebConsole', + 'scrapy.management.telnet.TelnetConsole', + 'scrapy.contrib.webconsole.enginestatus.EngineStatus', + 'scrapy.contrib.webconsole.stats.StatsDump', + 'scrapy.contrib.debug.StackTraceDebug', + ] + + +Available, enabled and disabled extensions +========================================== + +Not all available extensions will be enabled. Some of them usually depend on a +particular setting. For example, the Cache extension is typically available but +disabled by default unless you the :setting:`CACHE2_DIR` setting is set. Both +enabled and disabled extension can be accessed through the +:ref:`ref-extension-manager`. + +Accessing enabled extensions +============================ + +Even though it's not usually needed, you can access extension objects through +the :ref:`ref-extension-manager` which is populated when extensions are loaded. +For example, to access the ``WebConsole`` extension:: + + from scrapy.extension import extensions + webconsole_extension = extensions.enabled['WebConsole'] + +.. seealso:: + + :ref:`ref-extension-manager`, for the complete Extension manager reference. + +Writing your own extension +========================== + +Writing your own extension is easy. Each extension is a single Python class +which doesn't need to implement any particular method. + +All extension initialization code must be performed in the class constructor +(``__init__`` method). If that method raises the :exception:`NotConfigured` +exception, the extension will be disabled. Otherwise, the extension will be +enabled. + +Let's take a look at the following example extension which just logs a message +everytime a domain/spider is opened and closed:: + + from pydispatch import dispatcher + from scrapy.core import signals + + class SpiderOpenCloseLogging(object): + + def __init__(self): + dispatcher.connect(self.domain_opened, signal=signals.domain_opened) + dispatcher.connect(self.domain_closed, signal=signals.domain_closed) + + def domain_opened(self, domain, spider): + log.msg("opened domain %s" % domain) + + def domain_closed(self, domain, spider): + log.msg("closed domain %s" % domain) + +Built-in extensions +=================== + +See :ref:`ref-extensions`. + diff --git a/scrapy/trunk/docs/topics/index.rst b/scrapy/trunk/docs/topics/index.rst index 138d0dab7..24d236454 100644 --- a/scrapy/trunk/docs/topics/index.rst +++ b/scrapy/trunk/docs/topics/index.rst @@ -17,4 +17,7 @@ This section introduces all key concepts of Scrapy. downloader-middleware spider-middleware settings + extensions + stats + webconsole robotstxt diff --git a/scrapy/trunk/docs/topics/stats.rst b/scrapy/trunk/docs/topics/stats.rst new file mode 100644 index 000000000..a6d52425a --- /dev/null +++ b/scrapy/trunk/docs/topics/stats.rst @@ -0,0 +1,8 @@ +.. _topics-stats: + +Stats +===== + +Scrapy provides built-in functionality for collecting stats in a tree-like +structure. + diff --git a/scrapy/trunk/docs/topics/webconsole.rst b/scrapy/trunk/docs/topics/webconsole.rst new file mode 100644 index 000000000..c525a121f --- /dev/null +++ b/scrapy/trunk/docs/topics/webconsole.rst @@ -0,0 +1,83 @@ +.. _topics-webconsole: + +=========== +Web Console +=========== + +Scrapy comes with a built-in web server for monitoring and controlling a Scrapy +running process. + +The web console is :ref:`built-in Scrapy extension ` which +comes enabled by default, but you can also disable it if you're running tight +on memory. + +For more information about this extension see :ref:`ref-extensions-webconsole`. + +Writing a web console extension +=============================== + +Writing a web console extension is similar to writing any other :ref:`Scrapy +extensions ` except that the extension class must: + +1. catch the ``scrapy.management.web.webconsole_discover_module`` signal, and + return itself in the handler. + +2. have the following two attributes: + +.. attribute:: webconsole_id + + The id by which the Scrapy web interface will known this extension, and + also the main dir under which this extension interface will work. For + example, assuming Scrapy web server is listening on + http://localhost:8000/ and the ``webconsole_id='extension1'`` the web + main page for the interface of that extension will be: + + http://localhost:8000/extension1/ + +.. attribute:: webconsole_name + + The name by which the Scrapy web server will know that extension. That name + will be displayed in the main web console index, as the text that links to + the extension main page. + +3. implement the following method: + +.. method:: webconsole_render(wc_request) + +``wc_request`` is a `twisted.web.http.Request`_ object with the HTTP request +sent to the web console. + +.. _twisted.web.http.Request: http://python.net/crew/mwh/apidocs/twisted.web.http.Request.html + +It must return a str with the web page to render, typically containing HTML +code. + +Example web console extension +============================= + +Here's an example of a simple web console extension that just displays a "Hello +world!" text:: + + from pydispatch import dispatcher + from scrapy.management.web import webconsole_discover_module + + class HelloWorldConsole(object): + webconsole_id = 'helloworld' + webconsole_name = 'Hello world' + + def __init__(self): + dispatcher.connect(self.webconsole_discover_module, signal=webconsole_discover_module) + + def webconsole_discover_module(self): + return self + + def webconsole_render(self, wc_request): + return "

Hello world!

" + +If you start Scrapy with the web console enabled on http://localhost:8000/ and +you access the URL: + + http://localhost:8000/helloworld/ + +You will see a page containing a big "Hello World!" text. +