7.2 KiB
Telnet Console
System Message: ERROR/3 (<stdin>, line 1)
Unknown directive type "currentmodule".
.. currentmodule:: scrapy.extensions.telnet
Scrapy comes with a built-in telnet console for inspecting and controlling a Scrapy running process. The telnet console is just a regular python shell running inside the Scrapy process, so you can do literally anything from it.
The telnet console is a :ref:`built-in Scrapy extension <topics-extensions-ref>` which comes enabled by default, but you can also disable it if you want. For more information about the extension itself see :ref:`topics-extensions-ref-telnetconsole`.
System Message: ERROR/3 (<stdin>, line 13); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 13); backlink
Unknown interpreted text role "ref".Warning
It is not secure to use telnet console via public networks, as telnet doesn't provide any transport-layer security. Having username/password authentication doesn't change that.
Intended usage is connecting to a running Scrapy spider locally (spider process and telnet client are on the same machine) or over a secure connection (VPN, SSH tunnel). Please avoid using telnet console over insecure connections, or disable it completely using :setting:`TELNETCONSOLE_ENABLED` option.
System Message: ERROR/3 (<stdin>, line 23); backlink
Unknown interpreted text role "setting".Note
This feature is not supported when :setting:`TWISTED_REACTOR_ENABLED` is False.
System Message: ERROR/3 (<stdin>, line 30); backlink
Unknown interpreted text role "setting".System Message: ERROR/3 (<stdin>, line 32)
Unknown directive type "seealso".
.. seealso:: :ref:`security-telnet`
System Message: ERROR/3 (<stdin>, line 34)
Unknown directive type "highlight".
.. highlight:: none
How to access the telnet console
The telnet console listens on the first available TCP port from the range defined in the :setting:`TELNETCONSOLE_PORT` setting, which defaults to [6023, 6073]. To access the console you need to type:
System Message: ERROR/3 (<stdin>, line 39); backlink
Unknown interpreted text role "setting".telnet localhost 6023 Trying localhost... Connected to localhost. Escape character is '^]'. Username: Password: >>>
By default, the username is scrapy and the password is autogenerated. The autogenerated password can be seen on Scrapy logs like the example below:
2018-10-16 14:35:21 [scrapy.extensions.telnet] INFO: Telnet Password: 16f92501e8a59326
The default username and password can be overridden by the settings :setting:`TELNETCONSOLE_USERNAME` and :setting:`TELNETCONSOLE_PASSWORD`.
System Message: ERROR/3 (<stdin>, line 56); backlink
Unknown interpreted text role "setting".System Message: ERROR/3 (<stdin>, line 56); backlink
Unknown interpreted text role "setting".Warning
Username and password provide only a limited protection, as telnet is not using secure transport - by default traffic is not encrypted even if username and password are set.
You need the telnet program which comes installed by default in Windows, and most Linux distros.
Available variables in the telnet console
The telnet console is like a regular Python shell running inside the Scrapy process, so you can do anything from it including importing new modules, etc.
However, the telnet console comes with some default variables defined for convenience:
| Shortcut | Description |
|---|---|
| crawler | the Scrapy Crawler (:class:`scrapy.crawler.Crawler` object) System Message: ERROR/3 (<stdin>, line 82); backlink Unknown interpreted text role "class". |
| engine | Crawler.engine attribute |
| spider | the active spider |
| extensions | the Extension Manager (Crawler.extensions attribute) |
| stats | the Stats Collector (Crawler.stats attribute) |
| settings | the Scrapy settings object (Crawler.settings attribute) |
| est | print a report of the engine status |
| prefs | for memory debugging (see :ref:`topics-leaks`) System Message: ERROR/3 (<stdin>, line 96); backlink Unknown interpreted text role "ref". |
| p | a shortcut to the :func:`pprint.pprint` function System Message: ERROR/3 (<stdin>, line 98); backlink Unknown interpreted text role "func". |
Telnet console usage examples
Here are some example tasks you can do with the telnet console:
View engine status
You can use the est() method provided by the console to quickly show the engine status:
telnet localhost 6023 >>> est() Execution engine status time()-engine.start_time : 8.62972998619 len(engine.downloader.active) : 16 engine.scraper.is_idle() : False engine.spider.name : followall engine.spider_is_idle() : False engine._slot.closing : False len(engine._slot.inprogress) : 16 len(engine._slot.scheduler.dqs or []) : 0 len(engine._slot.scheduler.mqs) : 92 len(engine.scraper.slot.queue) : 0 len(engine.scraper.slot.active) : 0 engine.scraper.slot.active_size : 0 engine.scraper.slot.itemproc_size : 0 engine.scraper.slot.needs_backout() : False
Pause, resume and stop the Scrapy engine
To pause:
telnet localhost 6023 >>> engine.pause() >>>
To resume:
telnet localhost 6023 >>> engine.unpause() >>>
To stop:
telnet localhost 6023 >>> engine.stop() Connection closed by foreign host.
Telnet Console signals
System Message: ERROR/3 (<stdin>, line 159)
Unknown directive type "signal".
.. signal:: update_telnet_vars
System Message: ERROR/3 (<stdin>, line 160)
Unknown directive type "function".
.. function:: update_telnet_vars(telnet_vars)
Sent just before the telnet console is opened. You can hook up to this
signal to add, remove or update the variables that will be available in the
telnet local namespace. In order to do that, you need to update the
``telnet_vars`` dict in your handler.
:param telnet_vars: the dict of telnet variables
:type telnet_vars: dict
Telnet settings
These are the settings that control the telnet console's behaviour:
System Message: ERROR/3 (<stdin>, line 175)
Unknown directive type "setting".
.. setting:: TELNETCONSOLE_PORT
TELNETCONSOLE_PORT
Default: [6023, 6073]
The port range to use for the telnet console. If set to None, a dynamically assigned port is used.
System Message: ERROR/3 (<stdin>, line 186)
Unknown directive type "setting".
.. setting:: TELNETCONSOLE_HOST
TELNETCONSOLE_HOST
Default: '127.0.0.1'
The interface the telnet console should listen on
System Message: ERROR/3 (<stdin>, line 195)
Unknown directive type "seealso".
.. seealso:: :ref:`security-telnet`
System Message: ERROR/3 (<stdin>, line 198)
Unknown directive type "setting".
.. setting:: TELNETCONSOLE_USERNAME
TELNETCONSOLE_USERNAME
Default: 'scrapy'
The username used for the telnet console
System Message: ERROR/3 (<stdin>, line 208)
Unknown directive type "setting".
.. setting:: TELNETCONSOLE_PASSWORD
TELNETCONSOLE_PASSWORD
Default: None
The password used for the telnet console, default behaviour is to have it autogenerated