diff --git a/docs/topics/download-handlers.rst b/docs/topics/download-handlers.rst index 888bfaf08..c901e01e4 100644 --- a/docs/topics/download-handlers.rst +++ b/docs/topics/download-handlers.rst @@ -39,6 +39,10 @@ for additional schemes and to replace or disable default ones: "sftp": "my.download_handlers.SftpHandler", } +.. seealso:: :ref:`security-unencrypted-protocols` and + :ref:`security-local-resources`, for the security implications of the + default ``http``, ``ftp``, ``file`` and ``data`` handlers. + Replacing HTTP(S) download handlers ----------------------------------- diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst index 0c1af5276..5649453b1 100644 --- a/docs/topics/downloader-middleware.rst +++ b/docs/topics/downloader-middleware.rst @@ -379,6 +379,8 @@ that this risks leaking credentials to unrelated domains. This setting must be explicitly configured whenever :setting:`HTTPAUTH_USER` or :setting:`HTTPAUTH_PASS` is set. +.. seealso:: :ref:`security-credential-leakage` + .. _Basic access authentication: https://en.wikipedia.org/wiki/Basic_access_authentication diff --git a/docs/topics/security.rst b/docs/topics/security.rst index b013c4467..2ca270045 100644 --- a/docs/topics/security.rst +++ b/docs/topics/security.rst @@ -22,6 +22,8 @@ how to harden them along with the trade-offs involved. sources, whether the machine running Scrapy is exposed to a network you do not control, whether the data you handle is sensitive, and so on. +.. _security-untrusted-responses: + Treat responses as untrusted input ================================== @@ -37,6 +39,8 @@ derived from the response itself. TLS connections =============== +.. _security-certificate-verification: + Certificate verification ------------------------ @@ -62,6 +66,8 @@ man-in-the-middle attacks), set: * **Con:** you can no longer scrape sites with misconfigured certificates without re-disabling verification for them. +.. _security-tls-protocols-ciphers: + Protocol versions and ciphers ----------------------------- @@ -85,6 +91,8 @@ ciphers: * **Con:** you can no longer connect to servers that only support the excluded ciphers. +.. _security-unencrypted-protocols: + Unencrypted protocols ===================== @@ -113,6 +121,8 @@ Note that disabling the ``http`` handler also prevents plain-HTTP requests that result from following an ``http://`` redirect or link, which is often the point of disabling it. +.. _security-local-resources: + Local and non-network resources =============================== @@ -142,6 +152,8 @@ More generally, if you crawl URLs from untrusted sources, consider validating their schemes (and, where applicable, their hosts) before scheduling requests, to avoid server-side request forgery (SSRF) and similar issues. +.. _security-telnet: + Telnet console ============== @@ -175,6 +187,8 @@ If you do not use the telnet console, disable it entirely: * **Con:** you can no longer :ref:`inspect and control a running crawler ` through it. +.. _security-credential-leakage: + Credential leakage across domains ================================= diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst index b92733ede..11251a92c 100644 --- a/docs/topics/settings.rst +++ b/docs/topics/settings.rst @@ -774,6 +774,8 @@ implementation. handler `, so it's not guaranteed to be supported by all 3rd-party handlers. +.. seealso:: :ref:`security-tls-protocols-ciphers` + .. setting:: DOWNLOAD_TLS_MAX_VERSION DOWNLOAD_TLS_MAX_VERSION @@ -807,6 +809,8 @@ modern environments. by all 3rd-party handlers. Additionally, the set of supported TLS versions depends on the TLS implementation being used by the handler. +.. seealso:: :ref:`security-tls-protocols-ciphers` + .. setting:: DOWNLOAD_TLS_MIN_VERSION DOWNLOAD_TLS_MIN_VERSION @@ -819,6 +823,8 @@ be used by Scrapy. See :setting:`DOWNLOAD_TLS_MAX_VERSION` for the details and limitations. +.. seealso:: :ref:`security-tls-protocols-ciphers` + .. setting:: DOWNLOADER_CLIENT_TLS_VERBOSE_LOGGING DOWNLOADER_CLIENT_TLS_VERBOSE_LOGGING @@ -979,6 +985,9 @@ enabled in your project. See :setting:`DOWNLOAD_HANDLERS_BASE` for example format. +.. seealso:: :ref:`security-unencrypted-protocols` and + :ref:`security-local-resources` + .. setting:: DOWNLOAD_HANDLERS_BASE DOWNLOAD_HANDLERS_BASE @@ -1026,6 +1035,9 @@ handler (without replacement), place this in your ``settings.py``: "ftp": None, } +.. seealso:: :ref:`security-unencrypted-protocols` and + :ref:`security-local-resources` + .. setting:: DOWNLOAD_SLOTS @@ -1184,6 +1196,8 @@ when making a request and abort the request if the verification fails. certificate problems are logged when this setting is set to ``False``) depends on its implementation. +.. seealso:: :ref:`security-certificate-verification` + .. setting:: DUPEFILTER_CLASS DUPEFILTER_CLASS @@ -2074,6 +2088,8 @@ Default: ``True`` (``False`` when :setting:`TWISTED_REACTOR_ENABLED` is ``False` A boolean which specifies if the :ref:`telnet console ` will be enabled (provided its extension is also enabled). +.. seealso:: :ref:`security-telnet` + .. setting:: TEMPLATES_DIR TEMPLATES_DIR diff --git a/docs/topics/spider-middleware.rst b/docs/topics/spider-middleware.rst index 99bbdf292..e2cd0f986 100644 --- a/docs/topics/spider-middleware.rst +++ b/docs/topics/spider-middleware.rst @@ -355,6 +355,8 @@ Default: ``'scrapy.spidermiddlewares.referer.DefaultReferrerPolicy'`` using the special ``"referrer_policy"`` :ref:`Request.meta ` key, with the same acceptable values as for the ``REFERRER_POLICY`` setting. +.. seealso:: :ref:`security-credential-leakage` + Acceptable values for REFERRER_POLICY ************************************* diff --git a/docs/topics/telnetconsole.rst b/docs/topics/telnetconsole.rst index e274edc9b..a30258112 100644 --- a/docs/topics/telnetconsole.rst +++ b/docs/topics/telnetconsole.rst @@ -29,6 +29,8 @@ disable it if you want. For more information about the extension itself see .. note:: This feature is not supported when :setting:`TWISTED_REACTOR_ENABLED` is ``False``. +.. seealso:: :ref:`security-telnet` + .. highlight:: none How to access the telnet console @@ -190,6 +192,8 @@ Default: ``'127.0.0.1'`` The interface the telnet console should listen on +.. seealso:: :ref:`security-telnet` + .. setting:: TELNETCONSOLE_USERNAME