5.2 KiB
Telnet Console
System Message: ERROR/3 (<stdin>, line 7)
Unknown directive type "module".
.. module:: scrapy.management.telnet :synopsis: The Telnet Console
Scrapy comes with a built-in telnet console for inspecting and controlling a Scrapy running process. The telnet console is just a regular python shell running inside the Scrapy process, so you can do literally anything from it.
The telnet console is a :ref:`built-in Scrapy extension <topics-extensions-ref>` which comes enabled by default, but you can also disable it if you want. For more information about the extension itself see :ref:`topics-extensions-ref-telnetconsole`.
System Message: ERROR/3 (<stdin>, line 14); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 14); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 19)
Unknown directive type "highlight".
.. highlight:: none
How to access the telnet console
The telnet console listens in the TCP port defined in the :setting:`TELNETCONSOLE_PORT` setting, which defaults to 6023. To access the console you need to type:
System Message: ERROR/3 (<stdin>, line 24); backlink
Unknown interpreted text role "setting".telnet localhost 6023 >>>
You need the telnet program which comes installed by default in Windows, and most Linux distros.
Available aliases in the telnet console
The telnet console is like a regular Python shell running inside the Scrapy process, so you can do anything from it including imports, etc.
However, the telnet console comes with some default shortcuts defined for convenience:
| Shortcut | Description |
|---|---|
| engine | the Scrapy engine object (scrapy.core.engine.scrapyengine) |
| manager | the Scrapy manager object (scrapy.core.manager.scrapymanager) |
| extensions | the extensions object (scrapy.extension.extensions) |
| stats | the Scrapy stats object (scrapy.stats.stats) |
| settings | the Scrapy settings object (scrapy.conf.settings) |
| est | print a report of the current engine status |
| prefs | for memory debugging (see :ref:`topics-leaks`) System Message: ERROR/3 (<stdin>, line 59); backlink Unknown interpreted text role "ref". |
| p | a shortcut to the pprint.pprint function |
| hpy | for memory debugging (see :ref:`topics-leaks`) System Message: ERROR/3 (<stdin>, line 63); backlink Unknown interpreted text role "ref". |
Some example of using the telnet console
Here are some example tasks you can do with the telnet console:
View engine status
You can use the st() method of the Scrapy engine to quickly show its state using the telnet console:
telnet localhost 6023 >>> est() Execution engine status datetime.now()-self.start_time : 0:00:09.051588 self.is_idle() : False self.scheduler.is_idle() : False len(self.scheduler.pending_requests) : 1 self.downloader.is_idle() : False len(self.downloader.sites) : 1 self.downloader.has_capacity() : True self.pipeline.is_idle() : False len(self.pipeline.domaininfo) : 1 len(self._scraping) : 1 example.com self.domain_is_idle(domain) : False self.closing.get(domain) : None self.scheduler.domain_has_pending_requests(domain) : True len(self.scheduler.pending_requests[domain]) : 97 len(self.downloader.sites[domain].queue) : 17 len(self.downloader.sites[domain].active) : 25 len(self.downloader.sites[domain].transferring) : 8 self.downloader.sites[domain].closing : False self.downloader.sites[domain].lastseen : 2009-06-23 15:20:16.563675 self.pipeline.domain_is_idle(domain) : True len(self.pipeline.domaininfo[domain]) : 0 len(self._scraping[domain]) : 0
Pause, resume and stop Scrapy engine
To pause:
telnet localhost 6023 >>> engine.pause() >>>
To resume:
telnet localhost 6023 >>> engine.unpause() >>>
To stop:
telnet localhost 6023 >>> engine.stop() Connection closed by foreign host.