7.5 KiB
Built-in extensions reference
This document explains all extensions that come with Scrapy. For information on how to use them and how to write your own extensions, see the :ref:`extensions usage guide <topics-extensions>`.
System Message: ERROR/3 (<stdin>, line 7); backlink
Unknown interpreted text role "ref".General purpose extensions
Core Stats extension
System Message: ERROR/3 (<stdin>, line 18)
Unknown directive type "module".
.. module:: scrapy.stats.corestats :synopsis: Core stats collection
Enable the collection of core statistics, provided the stats collection are enabled (see :ref:`topics-stats`).
System Message: ERROR/3 (<stdin>, line 23); backlink
Unknown interpreted text role "ref".Web console extension
System Message: ERROR/3 (<stdin>, line 31)
Unknown directive type "module".
.. module:: scrapy.management.web :synopsis: Web management console
Provides an extensible web server for managing a Scrapy process. It's enabled by the :setting:`WEBCONSOLE_ENABLED` setting. The server will listen in the port specified in :setting:`WEBCONSOLE_PORT`, and will log to the file specified in :setting:`WEBCONSOLE_LOGFILE`.
System Message: ERROR/3 (<stdin>, line 36); backlink
Unknown interpreted text role "setting".System Message: ERROR/3 (<stdin>, line 36); backlink
Unknown interpreted text role "setting".System Message: ERROR/3 (<stdin>, line 36); backlink
Unknown interpreted text role "setting".The web server is designed to be extended by other extensions which can add their own management web interfaces.
See also :ref:`topics-webconsole` for information on how to write your own web console extension, and "Web console extensions" below for a list of available built-in (web console) extensions.
System Message: ERROR/3 (<stdin>, line 44); backlink
Unknown interpreted text role "ref".Telnet console extension
System Message: ERROR/3 (<stdin>, line 53)
Unknown directive type "module".
.. module:: scrapy.management.telnet :synopsis: Telnet management console
Provides a telnet console for getting into a Python interpreter inside the currently running Scrapy process, which can be very useful for debugging.
The telnet console must be enabled by the :setting:`TELNETCONSOLE_ENABLED` setting, and the server will listen in the port specified in :setting:`WEBCONSOLE_PORT`.
System Message: ERROR/3 (<stdin>, line 61); backlink
Unknown interpreted text role "setting".System Message: ERROR/3 (<stdin>, line 61); backlink
Unknown interpreted text role "setting".Spider reloader extension
System Message: ERROR/3 (<stdin>, line 68)
Unknown directive type "module".
.. module:: scrapy.contrib.spider.reloader :synopsis: Spider reloader extension
Reload spider objects once they've finished scraping, to release the resources and references to other objects they may hold.
Memory usage extension
System Message: ERROR/3 (<stdin>, line 81)
Unknown directive type "module".
.. module:: scrapy.contrib.memusage :synopsis: Memory usage extension
Allows monitoring the memory used by a Scrapy process and:
1, send a notification email when it exceeds a certain value 2. terminate the Scrapy process when it exceeds a certain value
The notification emails can be triggered when a certain warning value is reached (:setting:`MEMUSAGE_WARNING_MB`) and when the maximum value is reached (:setting:`MEMUSAGE_LIMIT_MB`) which will also cause the Scrapy process to be terminated.
System Message: ERROR/3 (<stdin>, line 91); backlink
Unknown interpreted text role "setting".System Message: ERROR/3 (<stdin>, line 91); backlink
Unknown interpreted text role "setting".This extension is enabled by the :setting:`MEMUSAGE_ENABLED` setting and can be configured with the following settings:
System Message: ERROR/3 (<stdin>, line 96); backlink
Unknown interpreted text role "setting".-
System Message: ERROR/3 (<stdin>, line 99); backlink
Unknown interpreted text role "setting".
:setting:`MEMUSAGE_WARNING_MB`
System Message: ERROR/3 (<stdin>, line 100); backlink
Unknown interpreted text role "setting".
:setting:`MEMUSAGE_NOTIFY_MAIL`
System Message: ERROR/3 (<stdin>, line 101); backlink
Unknown interpreted text role "setting".
-
System Message: ERROR/3 (<stdin>, line 102); backlink
Unknown interpreted text role "setting".
Memory debugger extension
System Message: ERROR/3 (<stdin>, line 107)
Unknown directive type "module".
.. module:: scrapy.contrib.memdebug :synopsis: Memory debugger extension
A memory debugger which collects some info about objects uncollected by the garbage collector and libxml2 memory leaks. To enable this extension turn on the :setting:`MEMDEBUG_ENABLED` setting. The report will be printed to standard output. If the :setting:`MEMDEBUG_NOTIFY` setting contains a list of emails the report will also be sent to those addresses.
System Message: ERROR/3 (<stdin>, line 112); backlink
Unknown interpreted text role "setting".System Message: ERROR/3 (<stdin>, line 112); backlink
Unknown interpreted text role "setting".Close domain extension
System Message: ERROR/3 (<stdin>, line 121)
Unknown directive type "module".
.. module:: scrapy.contrib.closedomain :synopsis: Close domain extension
Closes a domain/spider automatically when some conditions are met, using a specific closing reason for each condition.
The conditions for closing a domain can be configured through the following settings. Other conditions will be supported in the future.
System Message: ERROR/3 (<stdin>, line 132)
Unknown directive type "setting".
.. setting:: CLOSEDOMAIN_TIMEOUT
CLOSEDOMAIN_TIMEOUT
Default: 0
An integer which specifies a number of seconds. If the domain remains open for more than that number of second, it will be automatically closed with the reason closedomain_timeout. If zero (or non set) domains won't be closed by timeout.
System Message: ERROR/3 (<stdin>, line 144)
Unknown directive type "setting".
.. setting:: CLOSEDOMAIN_ITEMPASSED
CLOSEDOMAIN_ITEMPASSED
Default: 0
An integer which specifies a number of items. If the spider scrapes more than that amount if items and those items are passed by the item pipeline, the domain will be closed with the reason closedomain_itempassed. If zero (or non set) domains won't be closed by number of passed items.
Stack trace dump extension
System Message: ERROR/3 (<stdin>, line 159)
Unknown directive type "module".
.. module:: scrapy.contrib.debug :synopsis: Extensions for debugging Scrapy
Adds a SIGUSR1 signal handler which dumps the stack trace of a runnning Scrapy process when a SIGUSR1 signal is catched. After the stack trace is dumped, the Scrapy process continues to run normally.
The stack trace is sent to standard output, or to the Scrapy log file if :setting:`LOG_STDOUT` is enabled.
System Message: ERROR/3 (<stdin>, line 168); backlink
Unknown interpreted text role "setting".This extension only works on POSIX-compliant platforms (ie. not Windows).
StatsMailer extension
System Message: ERROR/3 (<stdin>, line 178)
Unknown directive type "module".
.. module:: scrapy.contrib.statsmailer :synopsis: StatsMailer extension
This simple extension can be used to send a notification email every time a domain has finished scraping, including the Scrapy stats collected. The email will be sent to all recipients specified in the :setting:`STATSMAILER_RCPTS` setting.
System Message: ERROR/3 (<stdin>, line 183); backlink
Unknown interpreted text role "setting".Web console extensions
System Message: ERROR/3 (<stdin>, line 191)
Unknown directive type "module".
.. module:: scrapy.contrib.webconsole :synopsis: Contains most built-in web console extensions
Here is a list of built-in web console extensions. For clarity "web console extension" is abbreviated as "WC extension".
For more information see the see the :ref:`web console documentation <topics-webconsole>`.
System Message: ERROR/3 (<stdin>, line 197); backlink
Unknown interpreted text role "ref".Scheduler queue WC extension
System Message: ERROR/3 (<stdin>, line 203)
Unknown directive type "module".
.. module:: scrapy.contrib.webconsole.scheduler :synopsis: Scheduler queue web console extension
Display a list of all pending Requests in the Scheduler queue, grouped by domain/spider.
Spider live stats WC extension
System Message: ERROR/3 (<stdin>, line 214)
Unknown directive type "module".
.. module:: scrapy.contrib.webconsole.livestats :synopsis: Spider live stats web console extension
Display a table with stats of all spider crawled by the current Scrapy run, including:
- Number of items scraped
- Number of pages crawled
- Number of pending requests in the scheduler
- Number of pending requests in the downloader queue
- Number of requests currently being downloaded
Engine status WC extension
System Message: ERROR/3 (<stdin>, line 231)
Unknown directive type "module".
.. module:: scrapy.contrib.webconsole.enginestatus :synopsis: Engine status web console extension
Display the current status of the Scrapy Engine, which is just the output of the Scrapy engine getstatus() method.
Stats collector dump WC extension
System Message: ERROR/3 (<stdin>, line 242)
Unknown directive type "module".
.. module:: scrapy.contrib.webconsole.stats :synopsis: Stats dump web console extension
Display the stats collected so far by the stats collector.