mirror of https://github.com/scrapy/scrapy.git
217 lines
7.2 KiB
ReStructuredText
217 lines
7.2 KiB
ReStructuredText
.. currentmodule:: scrapy.extensions.telnet
|
|
|
|
.. _topics-telnetconsole:
|
|
|
|
==============
|
|
Telnet Console
|
|
==============
|
|
|
|
Scrapy comes with a built-in telnet console for inspecting and controlling a
|
|
Scrapy running process. The telnet console is just a regular python shell
|
|
running inside the Scrapy process, so you can do literally anything from it.
|
|
|
|
The telnet console is a :ref:`built-in Scrapy extension
|
|
<topics-extensions-ref>` which comes enabled by default, but you can also
|
|
disable it if you want. For more information about the extension itself see
|
|
:ref:`topics-extensions-ref-telnetconsole`.
|
|
|
|
.. warning::
|
|
It is not secure to use telnet console via public networks, as telnet
|
|
doesn't provide any transport-layer security. Having username/password
|
|
authentication doesn't change that.
|
|
|
|
Intended usage is connecting to a running Scrapy spider locally
|
|
(spider process and telnet client are on the same machine)
|
|
or over a secure connection (VPN, SSH tunnel).
|
|
Please avoid using telnet console over insecure connections,
|
|
or disable it completely using :setting:`TELNETCONSOLE_ENABLED` option.
|
|
|
|
.. note::
|
|
This feature is not supported when :setting:`TWISTED_REACTOR_ENABLED` is ``False``.
|
|
|
|
.. seealso:: :ref:`security-telnet`
|
|
|
|
.. highlight:: none
|
|
|
|
How to access the telnet console
|
|
================================
|
|
|
|
The telnet console listens on the first available TCP port from the range
|
|
defined in the :setting:`TELNETCONSOLE_PORT` setting, which defaults to
|
|
``[6023, 6073]``. To access the console you need to type::
|
|
|
|
telnet localhost 6023
|
|
Trying localhost...
|
|
Connected to localhost.
|
|
Escape character is '^]'.
|
|
Username:
|
|
Password:
|
|
>>>
|
|
|
|
By default, the username is ``scrapy`` and the password is autogenerated. The
|
|
autogenerated password can be seen on Scrapy logs like the example below::
|
|
|
|
2018-10-16 14:35:21 [scrapy.extensions.telnet] INFO: Telnet Password: 16f92501e8a59326
|
|
|
|
The default username and password can be overridden by the settings
|
|
:setting:`TELNETCONSOLE_USERNAME` and :setting:`TELNETCONSOLE_PASSWORD`.
|
|
|
|
.. warning::
|
|
Username and password provide only a limited protection, as telnet
|
|
is not using secure transport - by default traffic is not encrypted
|
|
even if username and password are set.
|
|
|
|
You need the telnet program which comes installed by default in Windows, and
|
|
most Linux distros.
|
|
|
|
.. _telnet-vars:
|
|
|
|
Available variables in the telnet console
|
|
=========================================
|
|
|
|
The telnet console is like a regular Python shell running inside the Scrapy
|
|
process, so you can do anything from it including importing new modules, etc.
|
|
|
|
However, the telnet console comes with some default variables defined for
|
|
convenience:
|
|
|
|
+----------------+-------------------------------------------------------------------+
|
|
| Shortcut | Description |
|
|
+================+===================================================================+
|
|
| ``crawler`` | the Scrapy Crawler (:class:`scrapy.crawler.Crawler` object) |
|
|
+----------------+-------------------------------------------------------------------+
|
|
| ``engine`` | Crawler.engine attribute |
|
|
+----------------+-------------------------------------------------------------------+
|
|
| ``spider`` | the active spider |
|
|
+----------------+-------------------------------------------------------------------+
|
|
| ``extensions`` | the Extension Manager (Crawler.extensions attribute) |
|
|
+----------------+-------------------------------------------------------------------+
|
|
| ``stats`` | the Stats Collector (Crawler.stats attribute) |
|
|
+----------------+-------------------------------------------------------------------+
|
|
| ``settings`` | the Scrapy settings object (Crawler.settings attribute) |
|
|
+----------------+-------------------------------------------------------------------+
|
|
| ``est`` | print a report of the engine status |
|
|
+----------------+-------------------------------------------------------------------+
|
|
| ``prefs`` | for memory debugging (see :ref:`topics-leaks`) |
|
|
+----------------+-------------------------------------------------------------------+
|
|
| ``p`` | a shortcut to the :func:`pprint.pprint` function |
|
|
+----------------+-------------------------------------------------------------------+
|
|
|
|
Telnet console usage examples
|
|
=============================
|
|
|
|
.. skip: start
|
|
|
|
Here are some example tasks you can do with the telnet console:
|
|
|
|
View engine status
|
|
------------------
|
|
|
|
You can use the ``est()`` method provided by the console to quickly show the
|
|
engine status::
|
|
|
|
telnet localhost 6023
|
|
>>> est()
|
|
Execution engine status
|
|
|
|
time()-engine.start_time : 8.62972998619
|
|
len(engine.downloader.active) : 16
|
|
engine.scraper.is_idle() : False
|
|
engine.spider.name : followall
|
|
engine.spider_is_idle() : False
|
|
engine._slot.closing : False
|
|
len(engine._slot.inprogress) : 16
|
|
len(engine._slot.scheduler.dqs or []) : 0
|
|
len(engine._slot.scheduler.mqs) : 92
|
|
len(engine.scraper.slot.queue) : 0
|
|
len(engine.scraper.slot.active) : 0
|
|
engine.scraper.slot.active_size : 0
|
|
engine.scraper.slot.itemproc_size : 0
|
|
engine.scraper.slot.needs_backout() : False
|
|
|
|
|
|
Pause, resume and stop the Scrapy engine
|
|
----------------------------------------
|
|
|
|
To pause::
|
|
|
|
telnet localhost 6023
|
|
>>> engine.pause()
|
|
>>>
|
|
|
|
To resume::
|
|
|
|
telnet localhost 6023
|
|
>>> engine.unpause()
|
|
>>>
|
|
|
|
To stop::
|
|
|
|
telnet localhost 6023
|
|
>>> engine.stop()
|
|
Connection closed by foreign host.
|
|
|
|
.. skip: end
|
|
|
|
Telnet Console signals
|
|
======================
|
|
|
|
.. signal:: update_telnet_vars
|
|
.. function:: update_telnet_vars(telnet_vars)
|
|
|
|
Sent just before the telnet console is opened. You can hook up to this
|
|
signal to add, remove or update the variables that will be available in the
|
|
telnet local namespace. In order to do that, you need to update the
|
|
``telnet_vars`` dict in your handler.
|
|
|
|
:param telnet_vars: the dict of telnet variables
|
|
:type telnet_vars: dict
|
|
|
|
Telnet settings
|
|
===============
|
|
|
|
These are the settings that control the telnet console's behaviour:
|
|
|
|
.. setting:: TELNETCONSOLE_PORT
|
|
|
|
TELNETCONSOLE_PORT
|
|
------------------
|
|
|
|
Default: ``[6023, 6073]``
|
|
|
|
The port range to use for the telnet console. If set to ``None``, a dynamically
|
|
assigned port is used.
|
|
|
|
|
|
.. setting:: TELNETCONSOLE_HOST
|
|
|
|
TELNETCONSOLE_HOST
|
|
------------------
|
|
|
|
Default: ``'127.0.0.1'``
|
|
|
|
The interface the telnet console should listen on
|
|
|
|
.. seealso:: :ref:`security-telnet`
|
|
|
|
|
|
.. setting:: TELNETCONSOLE_USERNAME
|
|
|
|
TELNETCONSOLE_USERNAME
|
|
----------------------
|
|
|
|
Default: ``'scrapy'``
|
|
|
|
The username used for the telnet console
|
|
|
|
|
|
.. setting:: TELNETCONSOLE_PASSWORD
|
|
|
|
TELNETCONSOLE_PASSWORD
|
|
----------------------
|
|
|
|
Default: ``None``
|
|
|
|
The password used for the telnet console, default behaviour is to have it
|
|
autogenerated
|