10 KiB
Security
Scrapy defaults are optimized for web scraping, not for the security posture that you might expect from software that handles untrusted input or runs in a shared or exposed environment. Some common security practices are unnecessary for many scraping use cases, and a few can even prevent valid ones (for example, sites that you must scrape may use misconfigured TLS certificates or serve content over unencrypted protocols).
This page highlights the Scrapy defaults that have security implications, so that you can make an informed decision about whether to keep them, and explains how to harden them along with the trade-offs involved.
Note
None of the options below are silver bullets. Which of them make sense depends on your threat model: whether the URLs you crawl come from trusted sources, whether the machine running Scrapy is exposed to a network you do not control, whether the data you handle is sensitive, and so on.
Treat responses as untrusted input
Regardless of any setting, remember that response data comes from servers you do not control, even when you trust the site you are crawling, as responses may be tampered with in transit or the server itself may be compromised.
Never pass response data to functions that can execute code or otherwise act on their input in an unsafe way, such as :func:`eval`, :func:`exec`, or :func:`pickle.loads`, and be careful when writing response data to paths derived from the response itself.
System Message: ERROR/3 (<stdin>, line 34); backlink
Unknown interpreted text role "func".System Message: ERROR/3 (<stdin>, line 34); backlink
Unknown interpreted text role "func".System Message: ERROR/3 (<stdin>, line 34); backlink
Unknown interpreted text role "func".Memory use when parsing responses
Parsing a response with :ref:`selectors <topics-selectors>` builds an in-memory tree of the whole response body, which takes several times as much memory as the body itself. Scrapy parses without the size limits that libxml2 applies by default, so the size of that tree is bound only by the size of the response, as controlled by :setting:`DOWNLOAD_MAXSIZE` (default: 1 GiB).
System Message: ERROR/3 (<stdin>, line 44); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 44); backlink
Unknown interpreted text role "setting".XML entities are left unresolved, so the tree stays proportional to the response body even for input crafted as an XML bomb. A server can still make a crawler allocate a lot of memory by returning a very large response, though, so if you know the size of the responses you care about, lower the limit:
System Message: WARNING/2 (<stdin>, line 57)
Cannot analyze code. Pygments package not found.
.. code-block:: python
DOWNLOAD_MAXSIZE = 32 * 1024 * 1024 # 32 MiB
- Pro: a server cannot make the crawler allocate more memory than the limit allows, whether by returning a large response or by crafting one that is expensive to parse.
- Con: you can no longer scrape sites that legitimately serve responses above the limit, as those responses are dropped.
Parser limits
The limits that libxml2 applies by default, such as 256 nesting levels and 10 MB per text node, can be restored by overriding :attr:`~scrapy.http.TextResponse.selector` in a response subclass and swapping responses in a :ref:`downloader middleware <topics-downloader-middleware>`:
System Message: ERROR/3 (<stdin>, line 73); backlink
Unknown interpreted text role "attr".System Message: ERROR/3 (<stdin>, line 73); backlink
Unknown interpreted text role "ref".System Message: WARNING/2 (<stdin>, line 78)
Cannot analyze code. Pygments package not found.
.. code-block:: python
from functools import cached_property
from scrapy import Selector
from scrapy.http import HtmlResponse
class LimitedHtmlResponse(HtmlResponse):
@cached_property
def selector(self):
return Selector(self, huge_tree=False)
class LimitedParsingMiddleware:
def process_response(self, request, response, spider):
if isinstance(response, HtmlResponse):
return response.replace(cls=LimitedHtmlResponse)
return response
Do the same with :class:`~scrapy.http.XmlResponse` if you also parse XML.
System Message: ERROR/3 (<stdin>, line 98); backlink
Unknown interpreted text role "class".These limits apply per node, so :setting:`DOWNLOAD_MAXSIZE` remains your bound on total memory: a response made of many small elements is parsed in full and uses as much memory either way.
System Message: ERROR/3 (<stdin>, line 100); backlink
Unknown interpreted text role "setting".- Pro: deeply nested responses, and responses with very large individual nodes, become cheaper to parse.
- Con: parsing stops at those limits without raising, so a legitimate page that exceeds them yields incomplete data and no error.
TLS connections
Certificate verification
By default Scrapy does not verify the TLS certificate of HTTPS servers, as controlled by the :setting:`DOWNLOAD_VERIFY_CERTIFICATES` setting (default: False).
System Message: ERROR/3 (<stdin>, line 118); backlink
Unknown interpreted text role "setting".This default favors reach over security: many sites that are otherwise fine to scrape have expired, self-signed, or otherwise invalid certificates, and verifying certificates would make requests to them fail.
If the integrity of the connection matters to you (for example, to detect man-in-the-middle attacks), set:
System Message: WARNING/2 (<stdin>, line 129)
Cannot analyze code. Pygments package not found.
.. code-block:: python
DOWNLOAD_VERIFY_CERTIFICATES = True
- Pro: requests to servers with invalid or untrusted certificates fail instead of silently succeeding, protecting you from some man-in-the-middle attacks.
- Con: you can no longer scrape sites with misconfigured certificates without re-disabling verification for them.
Protocol versions and ciphers
You can restrict the TLS protocol versions that Scrapy accepts through the :setting:`DOWNLOAD_TLS_MIN_VERSION` and :setting:`DOWNLOAD_TLS_MAX_VERSION` settings, e.g. to reject obsolete protocol versions.
System Message: ERROR/3 (<stdin>, line 145); backlink
Unknown interpreted text role "setting".System Message: ERROR/3 (<stdin>, line 145); backlink
Unknown interpreted text role "setting".By default Scrapy uses the OpenSSL DEFAULT cipher list (:setting:`DOWNLOADER_CLIENT_TLS_CIPHERS`), which favors compatibility and still allows some older, weaker ciphers. Set it to None to instead use the curated cipher list of the underlying TLS implementation (Twisted), which excludes weak ciphers:
System Message: ERROR/3 (<stdin>, line 149); backlink
Unknown interpreted text role "setting".System Message: WARNING/2 (<stdin>, line 155)
Cannot analyze code. Pygments package not found.
.. code-block:: python
DOWNLOADER_CLIENT_TLS_CIPHERS = None
- Pro: connections that would negotiate a weak cipher fail instead of succeeding.
- Con: you can no longer connect to servers that only support the excluded ciphers.
Unencrypted protocols
By default Scrapy enables download handlers for unencrypted protocols, namely http:// and ftp:// (see :setting:`DOWNLOAD_HANDLERS_BASE`). Data sent and received over these protocols, including any credentials, travels in plain text and can be read or modified by anyone on the network path.
System Message: ERROR/3 (<stdin>, line 170); backlink
Unknown interpreted text role "setting".If you only crawl over encrypted protocols, you can disable the unencrypted ones so that no request can accidentally be sent unencrypted:
System Message: WARNING/2 (<stdin>, line 178)
Cannot analyze code. Pygments package not found.
.. code-block:: python
DOWNLOAD_HANDLERS = {
"http": None,
"ftp": None,
}
- Pro: a misconfigured or maliciously-redirected request cannot leak data over an unencrypted connection, as such requests fail instead.
- Con: you can no longer crawl resources that are only available over those protocols.
Note that disabling the http handler also prevents plain-HTTP requests that result from following an http:// redirect or link, which is often the point of disabling it.
Local and non-network resources
By default Scrapy enables download handlers for the file:// and data: schemes (see :setting:`DOWNLOAD_HANDLERS_BASE`). The file:// handler reads arbitrary files from the local filesystem, limited only by the permissions of the process running Scrapy.
System Message: ERROR/3 (<stdin>, line 200); backlink
Unknown interpreted text role "setting".This is convenient (for example, to parse a local HTML file), but it is a risk if any of the URLs you schedule come from an untrusted source: a crafted file:///etc/passwd URL could read local files.
If you do not need them, disable these handlers:
System Message: WARNING/2 (<stdin>, line 211)
Cannot analyze code. Pygments package not found.
.. code-block:: python
DOWNLOAD_HANDLERS = {
"file": None,
"data": None,
}
- Pro: crawled URLs cannot be used to read local files or inline data.
- Con: you can no longer fetch file:// or data: URLs.
More generally, if you crawl URLs from untrusted sources, consider validating their schemes (and, where applicable, their hosts) before scheduling requests, to avoid server-side request forgery (SSRF) and similar issues.
Telnet console
Scrapy enables the :ref:`telnet console <topics-telnetconsole>` by default (:setting:`TELNETCONSOLE_ENABLED`). The telnet console is a Python shell running inside the Scrapy process, so anyone who can connect to it can run arbitrary code in that process.
System Message: ERROR/3 (<stdin>, line 231); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 231); backlink
Unknown interpreted text role "setting".By default the console binds to 127.0.0.1 (:setting:`TELNETCONSOLE_HOST`) and is protected by a username (:setting:`TELNETCONSOLE_USERNAME`, default scrapy) and an automatically generated password (:setting:`TELNETCONSOLE_PASSWORD`), so it is only reachable from the local machine.
System Message: ERROR/3 (<stdin>, line 236); backlink
Unknown interpreted text role "setting".System Message: ERROR/3 (<stdin>, line 236); backlink
Unknown interpreted text role "setting".System Message: ERROR/3 (<stdin>, line 236); backlink
Unknown interpreted text role "setting".Warning
Telnet does not provide any transport-layer security, so the username/password authentication does not protect the credentials or the session from anyone able to observe the traffic. Never expose the telnet console over an untrusted network by changing :setting:`TELNETCONSOLE_HOST` to a non-local address.
System Message: ERROR/3 (<stdin>, line 244); backlink
Unknown interpreted text role "setting".If you do not use the telnet console, disable it entirely:
System Message: WARNING/2 (<stdin>, line 252)
Cannot analyze code. Pygments package not found.
.. code-block:: python
TELNETCONSOLE_ENABLED = False
Pro: removes a local code-execution surface and one less listening port.
Con: you can no longer :ref:`inspect and control a running crawler <topics-telnetconsole>` through it.
System Message: ERROR/3 (<stdin>, line 258); backlink
Unknown interpreted text role "ref".
Credential leakage across domains
Some Scrapy features attach credentials or other sensitive headers to requests, and a crawl that spans multiple domains can leak them to unintended hosts:
HTTP authentication credentials set through :class:`~scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware` are only sent to the domain set in :setting:`HTTPAUTH_DOMAIN`. Leave this set to the intended domain rather than None so that credentials are not sent to every domain you crawl.
System Message: ERROR/3 (<stdin>, line 269); backlink
Unknown interpreted text role "class".
System Message: ERROR/3 (<stdin>, line 269); backlink
Unknown interpreted text role "setting".
The Referer header may disclose the URLs you crawl to other sites. The default :setting:`REFERRER_POLICY` already avoids sending the referrer from HTTPS to HTTP, but you can tighten it further (for example, to same-origin or no-referrer) if needed.
System Message: ERROR/3 (<stdin>, line 275); backlink
Unknown interpreted text role "setting".