4.1 KiB
Cookies
Scrapy keeps track of the cookies that websites set and sends them back on later requests to those websites, just like a web browser does. That is the job of :class:`~scrapy.downloadermiddlewares.cookies.CookiesMiddleware`, which is enabled by default.
System Message: ERROR/3 (<stdin>, line 8); backlink
Unknown interpreted text role "class".Setting cookies on a request
Use the cookies parameter of :class:`~scrapy.Request` to send cookies of your own, either as a dict:
System Message: ERROR/3 (<stdin>, line 21); backlink
Unknown interpreted text role "class".System Message: WARNING/2 (<stdin>, line 24)
Cannot analyze code. Pygments package not found.
.. code-block:: python
request = Request(
url="https://example.com",
cookies={"currency": "USD", "country": "UY"},
)
Or as a list of dicts, which also lets you set cookie attributes:
System Message: WARNING/2 (<stdin>, line 33)
Cannot analyze code. Pygments package not found.
.. code-block:: python
request = Request(
url="https://example.com",
cookies=[
{
"name": "currency",
"value": "USD",
"domain": "example.com",
"path": "/currency",
"secure": True,
},
],
)
Setting attributes is only useful if the cookies are stored for later requests, i.e. if :reqmeta:`dont_merge_cookies` is not enabled.
System Message: ERROR/3 (<stdin>, line 48); backlink
Unknown interpreted text role "reqmeta".Caution!
Cookies set through the Cookie header are not handled by :class:`~scrapy.downloadermiddlewares.cookies.CookiesMiddleware`, which drops that header.
System Message: ERROR/3 (<stdin>, line 51); backlink
Unknown interpreted text role "class".Caution!
When a cookie name or value is a byte sequence that is not UTF-8 encoded, the cookie is dropped and a warning is logged. See :ref:`topics-logging-advanced-customization` to customize the logging behavior.
System Message: ERROR/3 (<stdin>, line 55); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 61)
Unknown directive type "reqmeta".
.. reqmeta:: cookiejar
Multiple cookie sessions per spider
By default all requests share a single cookie jar (session). To use different ones, pass an identifier in the :reqmeta:`cookiejar` request meta key:
System Message: ERROR/3 (<stdin>, line 66); backlink
Unknown interpreted text role "reqmeta".System Message: WARNING/2 (<stdin>, line 70)
Cannot analyze code. Pygments package not found.
.. code-block:: python
for i, url in enumerate(urls):
yield Request(url, meta={"cookiejar": i}, callback=self.parse_page)
The :reqmeta:`cookiejar` meta key is not "sticky", so you need to keep passing it along on subsequent requests:
System Message: ERROR/3 (<stdin>, line 75); backlink
Unknown interpreted text role "reqmeta".System Message: WARNING/2 (<stdin>, line 78)
Cannot analyze code. Pygments package not found.
.. code-block:: python
def parse_page(self, response):
return Request(
"https://example.com/otherpage",
meta={"cookiejar": response.meta["cookiejar"]},
callback=self.parse_other_page,
)
System Message: ERROR/3 (<stdin>, line 88)
Unknown directive type "reqmeta".
.. reqmeta:: dont_merge_cookies
Skipping the cookie jar for a request
Set the :reqmeta:`dont_merge_cookies` request meta key to True to keep a request from touching the cookie jar in either direction: no stored cookie is sent with the request, and no cookie received in the response is stored. The cookies of the request itself are ignored as well.
System Message: ERROR/3 (<stdin>, line 93); backlink
Unknown interpreted text role "reqmeta".System Message: ERROR/3 (<stdin>, line 99)
Unknown directive type "setting".
.. setting:: COOKIES_ENABLED
COOKIES_ENABLED
Default: True
Whether to enable :class:`~scrapy.downloadermiddlewares.cookies.CookiesMiddleware`. If disabled, no cookies are sent to web servers.
System Message: ERROR/3 (<stdin>, line 106); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 110)
Unknown directive type "setting".
.. setting:: COOKIES_DEBUG
COOKIES_DEBUG
Default: False
If enabled, Scrapy logs all cookies sent in requests (i.e. the Cookie header) and all cookies received in responses (i.e. the Set-Cookie header):
2011-04-06 14:35:10-0300 [scrapy.core.engine] INFO: Spider opened
2011-04-06 14:35:10-0300 [scrapy.downloadermiddlewares.cookies] DEBUG: Sending cookies to: <GET http://www.diningcity.com/netherlands/index.html>
Cookie: clientlanguage_nl=en_EN
2011-04-06 14:35:14-0300 [scrapy.downloadermiddlewares.cookies] DEBUG: Received cookies from: <200 http://www.diningcity.com/netherlands/index.html>
Set-Cookie: JSESSIONID=B~FA4DC0C496C8762AE4F1A620EAB34F38; Path=/
Set-Cookie: ip_isocode=US
Set-Cookie: clientlanguage_nl=en_EN; Expires=Thu, 07-Apr-2011 21:21:34 GMT; Path=/
2011-04-06 14:49:50-0300 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.diningcity.com/netherlands/index.html> (referer: None)
[...]
CookiesMiddleware
System Message: ERROR/3 (<stdin>, line 135)
Unknown directive type "module".
.. module:: scrapy.downloadermiddlewares.cookies :synopsis: Cookies Downloader Middleware
System Message: ERROR/3 (<stdin>, line 138)
Unknown directive type "autoclass".
.. autoclass:: CookiesMiddleware