21 KiB
Available Settings
Here's a list of all available Scrapy settings, in alphabetical order, along with their default values and the scope where they apply.
The scope, where available, shows where the setting is being used, if it's tied to any particular component. In that case the module of that component will be shown, typically an extension, middleware or pipeline. It also means that the component must be enabled in order for the setting to have any effect.
System Message: ERROR/3 (<stdin>, line 14)
Unknown directive type "setting".
.. setting:: ADAPTORS_DEBUG
ADAPTORS_DEBUG
Default: False
Enable debug mode for adaptors.
System Message: ERROR/3 (<stdin>, line 23); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 25)
Unknown directive type "setting".
.. setting:: BOT_NAME
BOT_NAME
Default: scrapybot
The name of the bot implemented by this Scrapy project. This will be used to construct the User-Agent by default, and also for logging.
System Message: ERROR/3 (<stdin>, line 35)
Unknown directive type "setting".
.. setting:: BOT_VERSION
BOT_VERSION
Default: 1.0
The version of the bot implemented by this Scrapy project. This will be used to construct the User-Agent by default.
System Message: ERROR/3 (<stdin>, line 45)
Unknown directive type "setting".
.. setting:: HTTPCACHE_DIR
HTTPCACHE_DIR
Default: '' (empty string)
The directory to use for storing the (low-level) HTTP cache. If empty the HTTP cache will be disabled.
System Message: ERROR/3 (<stdin>, line 55)
Unknown directive type "setting".
.. setting:: HTTPCACHE_EXPIRATION_SECS
HTTPCACHE_EXPIRATION_SECS
Default: 0
Number of seconds to use for HTTP cache expiration. Requests that were cached before this time will be re-downloaded. If zero, cached requests will always expire. Negative numbers means requests will never expire.
System Message: ERROR/3 (<stdin>, line 66)
Unknown directive type "setting".
.. setting:: HTTPCACHE_IGNORE_MISSING
HTTPCACHE_IGNORE_MISSING
Default: False
If enabled, requests not found in the cache will be ignored instead of downloaded.
System Message: ERROR/3 (<stdin>, line 75)
Unknown directive type "setting".
.. setting:: HTTPCACHE_SECTORIZE
HTTPCACHE_SECTORIZE
Default: True
Whether to split HTTP cache storage in several dirs for performance.
System Message: ERROR/3 (<stdin>, line 84)
Unknown directive type "setting".
.. setting:: CLOSEDOMAIN_NOTIFY
CLOSEDOMAIN_NOTIFY
Default: [] Scope: scrapy.contrib.closedomain
A list of emails to notify if the domain has been automatically closed by timeout.
System Message: ERROR/3 (<stdin>, line 94)
Unknown directive type "setting".
.. setting:: CLOSEDOMAIN_TIMEOUT
CLOSEDOMAIN_TIMEOUT
Default: 0 Scope: scrapy.contrib.closedomain
A timeout (in secs) for automatically closing a spider. Spiders that remain open for more than this time will be automatically closed. If zero, the automatically closing is disabled.
System Message: ERROR/3 (<stdin>, line 106)
Unknown directive type "setting".
.. setting:: CLUSTER_LOGDIR
CLUSTER_LOGDIR
Default: '' (empty string)
The directory to use for cluster logging.
System Message: ERROR/3 (<stdin>, line 115)
Unknown directive type "setting".
.. setting:: CLUSTER_MASTER_CACHEFILE
CLUSTER_MASTER_CACHEFILE
Default: ''
The file to use for storing the state of the cluster master, before shotting down. And also used for restoring the state on start up. If not set, state won't be persisted.
System Message: ERROR/3 (<stdin>, line 126)
Unknown directive type "setting".
.. setting:: CLUSTER_MASTER_ENABLED
CLUSTER_MASTER_ENABLED
Default: False
A boolean which specifies whether to enabled the cluster master.
System Message: ERROR/3 (<stdin>, line 135)
Unknown directive type "setting".
.. setting:: CLUSTER_MASTER_NODES
CLUSTER_MASTER_NODES
Default: {}
A dict which defines the nodes of the cluster. The keys are the node/worker names and the values are the worker URLs.
Example:
CLUSTER_MASTER_NODES = {
'local': 'localhost:8789',
'remote': 'someworker.example.com:8789',
}
System Message: ERROR/3 (<stdin>, line 152)
Unknown directive type "setting".
.. setting:: CLUSTER_MASTER_POLL_INTERVAL
CLUSTER_MASTER_POLL_INTERVAL
Default: 60
The amount of time (in secs) that the master should wait before polling the workers.
System Message: ERROR/3 (<stdin>, line 162)
Unknown directive type "setting".
.. setting:: CLUSTER_MASTER_PORT
CLUSTER_MASTER_PORT
Default: 8790
The port where the cluster master will listen.
System Message: ERROR/3 (<stdin>, line 171)
Unknown directive type "setting".
.. setting:: CLUSTER_WORKER_ENABLED
CLUSTER_WORKER_ENABLED
Default: False
A boolean which specifies whether to enabled the cluster master.
System Message: ERROR/3 (<stdin>, line 180)
Unknown directive type "setting".
.. setting:: CLUSTER_WORKER_MAXPROC
CLUSTER_WORKER_MAXPROC
Default: 4
The maximum number of process that the cluster worker will be allowed to spawn.
System Message: ERROR/3 (<stdin>, line 189)
Unknown directive type "setting".
.. setting:: CLUSTER_WORKER_PORT
CLUSTER_WORKER_PORT
Default: 8789
The port where the cluster worker will listen.
System Message: ERROR/3 (<stdin>, line 198)
Unknown directive type "setting".
.. setting:: COMMANDS_MODULE
COMMANDS_MODULE
Default: '' (empty string)
A module to use for looking for custom Scrapy commands. This is used to add custom command for your Scrapy project.
Example:
COMMANDS_MODULE = 'mybot.commands'
System Message: ERROR/3 (<stdin>, line 212)
Unknown directive type "setting".
.. setting:: COMMANDS_SETTINGS_MODULE
COMMANDS_SETTINGS_MODULE
Default: '' (empty string)
A module to use for looking for custom Scrapy command settings.
Example:
COMMANDS_SETTINGS_MODULE = 'mybot.conf.commands'
System Message: ERROR/3 (<stdin>, line 225)
Unknown directive type "setting".
.. setting:: CONCURRENT_DOMAINS
CONCURRENT_DOMAINS
Default: 8
Number of domains to scrape concurrently in one process. This doesn't affect the number of domains scraped concurrently by the Scrapy cluster which spawns a new process per domain.
System Message: ERROR/3 (<stdin>, line 236)
Unknown directive type "setting".
.. setting:: COOKIES_DEBUG
DEFAULT_ITEM_CLASS
Default: 'scrapy.item.ScrapedItem'
The default class that will be used for instantiating items in the :ref:`the Scrapy shell <topics-shell>`.
System Message: ERROR/3 (<stdin>, line 252); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 255)
Unknown directive type "setting".
.. setting:: DEFAULT_SPIDER
DEFAULT_SPIDER
Default: None
The default spider class that will be instantiated for URLs for which no specific spider is found. This class must have a constructor which receives as only parameter the domain name of the given URL.
System Message: ERROR/3 (<stdin>, line 266)
Unknown directive type "setting".
.. setting:: DEPTH_LIMIT
DEPTH_LIMIT
Default: 0
The maximum depth that will be allowed to crawl for any site. If zero, no limit will be imposed.
System Message: ERROR/3 (<stdin>, line 276)
Unknown directive type "setting".
.. setting:: DEPTH_STATS
DEPTH_STATS
Default: True
Whether to collect depth stats.
System Message: ERROR/3 (<stdin>, line 285)
Unknown directive type "setting".
.. setting:: DOWNLOADER_DEBUG
DOWNLOADER_DEBUG
Default: False
Whether to enable the Downloader debugging mode.
System Message: ERROR/3 (<stdin>, line 294)
Unknown directive type "setting".
.. setting:: DOWNLOADER_MIDDLEWARES
DOWNLOADER_MIDDLEWARES
Default:: {}
A dict containing the downloader middlewares enabled in your project, and their orders. For more info see :ref:`topics-downloader-middleware-setting`.
System Message: ERROR/3 (<stdin>, line 301); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 304)
Unknown directive type "setting".
.. setting:: DOWNLOADER_MIDDLEWARES_BASE
DOWNLOADER_MIDDLEWARES_BASE
Default:
{
'scrapy.contrib.downloadermiddleware.robotstxt.RobotsTxtMiddleware': 100,
'scrapy.contrib.downloadermiddleware.errorpages.ErrorPagesMiddleware': 200,
'scrapy.contrib.downloadermiddleware.httpauth.HttpAuthMiddleware': 300,
'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware': 400,
'scrapy.contrib.downloadermiddleware.retry.RetryMiddleware': 500,
'scrapy.contrib.downloadermiddleware.common.CommonMiddleware': 550,
'scrapy.contrib.downloadermiddleware.redirect.RedirectMiddleware': 600,
'scrapy.contrib.downloadermiddleware.cookies.CookiesMiddleware': 700,
'scrapy.contrib.downloadermiddleware.httpcompression.HttpCompressionMiddleware': 800,
'scrapy.contrib.downloadermiddleware.debug.CrawlDebug': 840,
'scrapy.contrib.downloadermiddleware.stats.DownloaderStats': 850,
'scrapy.contrib.downloadermiddleware.httpcache.HttpCacheMiddleware': 900,
}
A dict containing the downloader middlewares enabled by default in Scrapy. You should never modify this setting in your project, modify :setting:`DOWNLOADER_MIDDLEWARES` instead. For more info see :ref:`topics-downloader-middleware-setting`.
System Message: ERROR/3 (<stdin>, line 326); backlink
Unknown interpreted text role "setting".System Message: ERROR/3 (<stdin>, line 326); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 331)
Unknown directive type "setting".
.. setting:: DOWNLOADER_STATS
DOWNLOADER_STATS
Default: True
Whether to enable downloader stats collection.
System Message: ERROR/3 (<stdin>, line 340)
Unknown directive type "setting".
.. setting:: DOWNLOAD_DELAY
DOWNLOAD_DELAY
Default: 0
The amount of time (in secs) that the downloader should wait before downloading consecutive pages from the same spider. This can be used to throttle the crawling speed to avoid hitting servers too hard. Decimal numbers are supported. Example:
DOWNLOAD_DELAY = 0.25 # 250 ms of delay
System Message: ERROR/3 (<stdin>, line 354)
Unknown directive type "setting".
.. setting:: DOWNLOAD_TIMEOUT
DOWNLOAD_TIMEOUT
Default: 180
The amount of time (in secs) that the downloader will wait before timing out.
System Message: ERROR/3 (<stdin>, line 363)
Unknown directive type "setting".
.. setting:: DUPEFILTER_FILTERCLASS
DUPEFILTER_FILTERCLASS
Default: scrapy.contrib.spidermiddleware.SimplePerDomainFilter
The class used to detect and filter duplicated requests.
Default SimplePerDomainFilter filter based on request fingerprint and grouping per domain.
System Message: ERROR/3 (<stdin>, line 375)
Unknown directive type "setting".
.. setting:: ENGINE_DEBUG
ENGINE_DEBUG
Default: False
Whether to enable the Scrapy Engine debugging mode.
System Message: ERROR/3 (<stdin>, line 384)
Unknown directive type "setting".
.. setting:: ENABLED_SPIDERS_FILE
ENABLED_SPIDERS_FILE
Default: '' (empty string)
The path to a file containing a list of spiders (one domain name per line). Those spiders will be considered enabled by Scrapy, and will be the spiders crawled automatically when running scrapy-ctl.py crawl with no arguments.
If this setting is unset, all spiders to crawl must be passed explicitly in the crawl command.
Example:
'/etc/mybot/enabled_spiders.list'
System Message: ERROR/3 (<stdin>, line 402)
Unknown directive type "setting".
.. setting:: EXTENSIONS
EXTENSIONS
Default:
[
'scrapy.stats.corestats.CoreStats',
'scrapy.xpath.extension.ResponseLibxml2',
'scrapy.management.web.WebConsole',
'scrapy.management.telnet.TelnetConsole',
'scrapy.contrib.webconsole.scheduler.SchedulerQueue',
'scrapy.contrib.webconsole.livestats.LiveStats',
'scrapy.contrib.webconsole.spiderctl.Spiderctl',
'scrapy.contrib.webconsole.enginestatus.EngineStatus',
'scrapy.contrib.webconsole.stats.StatsDump',
'scrapy.contrib.webconsole.spiderstats.SpiderStats',
'scrapy.contrib.spider.reloader.SpiderReloader',
'scrapy.contrib.memusage.MemoryUsage',
'scrapy.contrib.memdebug.MemoryDebugger',
'scrapy.contrib.closedomain.CloseDomain',
'scrapy.contrib.debug.StackTraceDump',
'scrapy.contrib.response.soup.ResponseSoup',
]
The list of available extensions. Keep in mind that some of them need need to be enabled through a setting. By default, this setting contains all stable built-in extensions.
For more information See the :ref:`extensions user guide <topics-extensions>` and the :ref:`list of available extensions <ref-extensions>`.
System Message: ERROR/3 (<stdin>, line 432); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 432); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 435)
Unknown directive type "setting".
.. setting:: GROUPSETTINGS_ENABLED
GROUPSETTINGS_ENABLED
Default: False
Whether to enable group settings where spiders pull their settings from.
System Message: ERROR/3 (<stdin>, line 444)
Unknown directive type "setting".
.. setting:: GROUPSETTINGS_MODULE
GROUPSETTINGS_MODULE
Default: '' (empty string)
The module to use for pulling settings from, if the group settings is enabled.
System Message: ERROR/3 (<stdin>, line 453)
Unknown directive type "setting".
.. setting:: ITEM_PIPELINES
ITEM_PIPELINES
Default: []
The item pipelines to use (a list of classes).
Example:
ITEM_PIPELINES = [
'mybot.pipeline.validate.ValidateMyItem',
'mybot.pipeline.validate.StoreMyItem'
]
System Message: ERROR/3 (<stdin>, line 469)
Unknown directive type "setting".
.. setting:: LOG_ENABLED
LOG_ENABLED
Default: True
Enable logging.
System Message: ERROR/3 (<stdin>, line 478)
Unknown directive type "setting".
.. setting:: LOG_STDOUT
LOG_STDOUT
Default: False
If enabled logging will be sent to standard output, otherwise standard error will be used.
System Message: ERROR/3 (<stdin>, line 488)
Unknown directive type "setting".
.. setting:: LOGFILE
LOGFILE
Default: None
File name to use for logging output. If None, standard input (or error) will be used depending on the value of the LOG_STDOUT setting.
System Message: ERROR/3 (<stdin>, line 498)
Unknown directive type "setting".
.. setting:: LOGLEVEL
LOGLEVEL
Default: 'DEBUG'
Minimum level to log. Available levels are: SILENT, CRITICAL, ERROR, WARNING, INFO, DEBUG, TRACE
System Message: ERROR/3 (<stdin>, line 508)
Unknown directive type "setting".
.. setting:: MAIL_FROM
MAIL_FROM
Default: 'scrapy@localhost'
Email to use as sender address for sending emails using the :ref:`Scrapy e-mail sending facility <ref-email>`.
System Message: ERROR/3 (<stdin>, line 515); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 518)
Unknown directive type "setting".
.. setting:: MAIL_HOST
MAIL_HOST
Default: 'localhost'
Host to use for sending emails using the :ref:`Scrapy e-mail sending facility <ref-email>`.
System Message: ERROR/3 (<stdin>, line 525); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 528)
Unknown directive type "setting".
.. setting:: MEMDEBUG_ENABLED
MEMDEBUG_ENABLED
Default: False
Whether to enable memory debugging.
System Message: ERROR/3 (<stdin>, line 537)
Unknown directive type "setting".
.. setting:: MEMDEBUG_NOTIFY
MEMDEBUG_NOTIFY
Default: []
When memory debugging is enabled a memory report will be sent to the specified addresses if this setting is not empty, otherwise the report will be written to the log.
Example:
MEMDEBUG_NOTIFY = ['user@example.com']
System Message: ERROR/3 (<stdin>, line 552)
Unknown directive type "setting".
.. setting:: MEMUSAGE_ENABLED
MEMUSAGE_ENABLED
Default: False
Scope: scrapy.contrib.memusage
Whether to enable the memory usage extension that will shutdown the Scrapy process when it exceeds a memory limit, and also notify by email when that happened.
See :ref:`ref-extensions-memusage`.
System Message: ERROR/3 (<stdin>, line 565); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 567)
Unknown directive type "setting".
.. setting:: MEMUSAGE_LIMIT_MB
MEMUSAGE_LIMIT_MB
Default: 0
Scope: scrapy.contrib.memusage
The maximum amount of memory to allow (in megabytes) before shutting down Scrapy (if MEMUSAGE_ENABLED is True). If zero, no check will be performed.
See :ref:`ref-extensions-memusage`.
System Message: ERROR/3 (<stdin>, line 579); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 581)
Unknown directive type "setting".
.. setting:: MEMUSAGE_NOTIFY_MAIL
MEMUSAGE_NOTIFY_MAIL
Default: False
Scope: scrapy.contrib.memusage
A list of emails to notify if the memory limit has been reached.
Example:
MEMUSAGE_NOTIFY_MAIL = ['user@example.com']
See :ref:`ref-extensions-memusage`.
System Message: ERROR/3 (<stdin>, line 596); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 598)
Unknown directive type "setting".
.. setting:: MEMUSAGE_REPORT
MEMUSAGE_REPORT
Default: False
Scope: scrapy.contrib.memusage
Whether to send a memory usage report after each domain has been closed.
See :ref:`ref-extensions-memusage`.
System Message: ERROR/3 (<stdin>, line 609); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 611)
Unknown directive type "setting".
.. setting:: MEMUSAGE_WARNING_MB
MEMUSAGE_WARNING_MB
Default: 0
Scope: scrapy.contrib.memusage
The maximum amount of memory to allow (in megabytes) before sending a warning email notifying about it. If zero, no warning will be produced.
System Message: ERROR/3 (<stdin>, line 623)
Unknown directive type "setting".
.. setting:: MYSQL_CONNECTION_SETTINGS
MYSQL_CONNECTION_SETTINGS
Default: {}
Scope: scrapy.utils.db.mysql_connect
Settings to use for MySQL connections performed through scrapy.utils.db.mysql_connect
System Message: ERROR/3 (<stdin>, line 635)
Unknown directive type "setting".
.. setting:: NEWSPIDER_MODULE
NEWSPIDER_MODULE
Default: ''
Module where to create new spiders using the genspider command.
Example:
NEWSPIDER_MODULE = 'mybot.spiders_dev'
System Message: ERROR/3 (<stdin>, line 648)
Unknown directive type "setting".
.. setting:: PROJECT_NAME
PROJECT_NAME
Default: Not Defined
The name of the current project. It matches the project module name as created by startproject command, and is only defined by project settings file.
System Message: ERROR/3 (<stdin>, line 658)
Unknown directive type "setting".
.. setting:: REDIRECT_MAX_TIMES
REDIRECT_MAX_TIMES
Default: 20
Defines the maximun times a request can be redirected. After this maximun the request's response is returned as is. We used Firefox default value for the same task.
System Message: ERROR/3 (<stdin>, line 669)
Unknown directive type "setting".
.. setting:: REDIRECT_MAX_METAREFRESH_DELAY
REDIRECT_MAX_METAREFRESH_DELAY
System Message: WARNING/2 (<stdin>, line 672)
Title underline too short.
REDIRECT_MAX_METAREFRESH_DELAY ---------------------------
Default: 100
Some sites use meta-refresh for redirecting to a session expired page, so we restrict automatic redirection to a maximum delay (in seconds)
System Message: ERROR/3 (<stdin>, line 679)
Unknown directive type "setting".
.. setting:: REQUEST_HEADER_ACCEPT
REQUEST_HEADER_ACCEPT
Default: 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'
Default value to use for the Accept request header (if not already set before).
See :ref:`ref-downloader-middleware-common`.
System Message: ERROR/3 (<stdin>, line 689); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 691)
Unknown directive type "setting".
.. setting:: REQUEST_HEADER_ACCEPT_LANGUAGE
REQUEST_HEADER_ACCEPT_LANGUAGE
Default: 'en'
Default value to use for the Accept-Language request header, if not already set before.
See :ref:`ref-downloader-middleware-common`.
System Message: ERROR/3 (<stdin>, line 701); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 703)
Unknown directive type "setting".
.. setting:: REQUESTS_QUEUE_SIZE
REQUESTS_PER_DOMAIN
Default: 8
Specifies how many concurrent (ie. simultaneous) requests will be performed per open spider.
REQUESTS_QUEUE_SIZE
Default: 0
Scope: scrapy.contrib.spidermiddleware.limit
If non zero, it will be used as an upper limit for the amount of requests that can be scheduled per domain.
System Message: ERROR/3 (<stdin>, line 723)
Unknown directive type "setting".
.. setting:: ROBOTSTXT_OBEY
ROBOTSTXT_OBEY
Default: False
Scope: scrapy.contrib.downloadermiddleware.robotstxt
If enabled, Scrapy will respect robots.txt policies. For more information see :topic:`robotstxt`
System Message: ERROR/3 (<stdin>, line 732); backlink
Unknown interpreted text role "topic".System Message: ERROR/3 (<stdin>, line 735)
Unknown directive type "setting".
.. setting:: SCHEDULER
SCHEDULER
Default: 'scrapy.core.scheduler.Scheduler'
The scheduler to use for crawling.
System Message: ERROR/3 (<stdin>, line 744)
Unknown directive type "setting".
.. setting:: SCHEDULER_ORDER
SCHEDULER_ORDER
Default: 'BFO'
Scope: scrapy.core.scheduler
The order to use for the crawling scheduler. Available orders are:
- 'BFO': Breadth-first order - typically consumes more memory but reaches most relevant pages earlier.
- 'DFO': Depth-first order - typically consumes less memory than but takes longer to reach most relevant pages.
System Message: ERROR/3 (<stdin>, line 764)
Unknown directive type "setting".
.. setting:: SCHEDULER_MIDDLEWARES
SCHEDULER_MIDDLEWARES
Default:: {}
A dict containing the scheduler middlewares enabled in your project, and their orders.
System Message: ERROR/3 (<stdin>, line 774)
Unknown directive type "setting".
.. setting:: SCHEDULER_MIDDLEWARES_BASE
SCHEDULER_MIDDLEWARES_BASE
Default:
[
'scrapy.contrib.schedulermiddleware.duplicatesfilter.DuplicatesFilterMiddleware',
]
A dict containing the scheduler middlewares enabled by default in Scrapy. You should never modify this setting in your project, modify :setting:`SCHEDULER_MIDDLEWARES` instead.
System Message: ERROR/3 (<stdin>, line 786); backlink
Unknown interpreted text role "setting".System Message: ERROR/3 (<stdin>, line 790)
Unknown directive type "setting".
.. setting:: SPIDERPROFILER_ENABLED
SPIDERPROFILER_ENABLED
Default: False
Enable the spider profiler. Warning: this could have a big impact in performance.
System Message: ERROR/3 (<stdin>, line 800)
Unknown directive type "setting".
.. setting:: SPIDER_MIDDLEWARES
SPIDER_MIDDLEWARES
Default:: {}
A dict containing the spider middlewares enabled in your project, and their orders. For more info see :ref:`topics-spider-middleware-setting`.
System Message: ERROR/3 (<stdin>, line 807); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 810)
Unknown directive type "setting".
.. setting:: SPIDER_MIDDLEWARES_BASE
SPIDER_MIDDLEWARES_BASE
Default:
{
'scrapy.contrib.itemsampler.ItemSamplerMiddleware': 100,
'scrapy.contrib.spidermiddleware.limit.RequestLimitMiddleware': 200,
'scrapy.contrib.spidermiddleware.restrict.RestrictMiddleware': 300,
'scrapy.contrib.spidermiddleware.offsite.OffsiteMiddleware': 500,
'scrapy.contrib.spidermiddleware.referer.RefererMiddleware': 700,
'scrapy.contrib.spidermiddleware.urllength.UrlLengthMiddleware': 800,
'scrapy.contrib.spidermiddleware.depth.DepthMiddleware': 900,
}
A dict containing the spider middlewares enabled by default in Scrapy. You should never modify this setting in your project, modify :setting:`SPIDER_MIDDLEWARES` instead. For more info see :ref:`topics-spider-middleware-setting`.
System Message: ERROR/3 (<stdin>, line 827); backlink
Unknown interpreted text role "setting".System Message: ERROR/3 (<stdin>, line 827); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 832)
Unknown directive type "setting".
.. setting:: SPIDER_MODULES
SPIDER_MODULES
Default: []
A list of modules where Scrapy will look for spiders.
Example:
SPIDER_MODULES = ['mybot.spiders_prod', 'mybot.spiders_dev']
System Message: ERROR/3 (<stdin>, line 845)
Unknown directive type "setting".
.. setting:: STATS_CLEANUP
STATS_CLEANUP
Default: False
Whether to cleanup (to save memory) the stats for a given domain, when the domain is closed.
System Message: ERROR/3 (<stdin>, line 855)
Unknown directive type "setting".
.. setting:: STATS_DEBUG
STATS_DEBUG
Default: False
Enable debugging mode for Scrapy stats. This logs the stats when a domain is closed.
System Message: ERROR/3 (<stdin>, line 865)
Unknown directive type "setting".
.. setting:: STATS_ENABLED
STATS_ENABLED
Default: True
Enable stats collection.
System Message: ERROR/3 (<stdin>, line 874)
Unknown directive type "setting".
.. setting:: TELNETCONSOLE_ENABLED
TELNETCONSOLE_ENABLED
Default: True
Scope: scrapy.management.telnet
A boolean which specifies if the telnet management console will be enabled (provided its extension is also enabled).
System Message: ERROR/3 (<stdin>, line 886)
Unknown directive type "setting".
.. setting:: TELNETCONSOLE_PORT
TELNETCONSOLE_PORT
Default: None
Scope: scrapy.management.telnet
The port to use for the telnet console. If unset, a dynamically assigned port is used.
System Message: ERROR/3 (<stdin>, line 899)
Unknown directive type "setting".
.. setting:: TEMPLATES_DIR
TEMPLATES_DIR
Default: templates dir inside scrapy module
The directory where to look for template when creating new projects with scrapy-admin.py newproject.
System Message: ERROR/3 (<stdin>, line 909)
Unknown directive type "setting".
.. setting:: URLLENGTH_LIMIT
URLLENGTH_LIMIT
Default: 2083
Scope: contrib.spidermiddleware.urllength
The maximum URL length to allow for crawled URLs. For more information about the default value for this setting see: http://www.boutell.com/newfaq/misc/urllength.html
System Message: ERROR/3 (<stdin>, line 921)
Unknown directive type "setting".
.. setting:: USER_AGENT
USER_AGENT
Default: "%s/%s" % (BOT_NAME, BOT_VERSION)
The default User-Agent to use when crawling, unless overrided.
System Message: ERROR/3 (<stdin>, line 930)
Unknown directive type "setting".
.. setting:: WEBCONSOLE_ENABLED
WEBCONSOLE_ENABLED
Default: True
A boolean which specifies if the web management console will be enabled (provided its extension is also enabled).
System Message: ERROR/3 (<stdin>, line 940)
Unknown directive type "setting".
.. setting:: WEBCONSOLE_LOGFILE
WEBCONSOLE_LOGFILE
Default: None
A file to use for logging HTTP requests made to the web console. If unset web the log is sent to standard scrapy log.
System Message: ERROR/3 (<stdin>, line 950)
Unknown directive type "setting".
.. setting:: WEBCONSOLE_PORT
WEBCONSOLE_PORT
Default: None
The port to use for the web console. If unset, a dynamically assigned port is used.