diff --git a/docs/faq.rst b/docs/faq.rst index 80658a5bf..ef3d07a2b 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -292,7 +292,7 @@ Does Scrapy manage cookies automatically? Yes, Scrapy receives and keeps track of cookies sent by servers, and sends them back on subsequent requests, like any regular web browser does. -For more info see :ref:`topics-request-response` and :ref:`cookies-mw`. +For more info see :ref:`cookies`. How can I see the cookies being sent and received from Scrapy? -------------------------------------------------------------- diff --git a/docs/index.rst b/docs/index.rst index d45b2c208..de06e3488 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -78,6 +78,7 @@ Basic concepts topics/item-pipeline topics/feed-exports topics/request-response + topics/cookies topics/link-extractors topics/settings topics/exceptions @@ -109,6 +110,9 @@ Basic concepts :doc:`topics/request-response` Understand the classes used to represent HTTP requests and responses. +:doc:`topics/cookies` + Send and receive cookies. + :doc:`topics/link-extractors` Convenient classes to extract links to follow from pages. diff --git a/docs/topics/cookies.rst b/docs/topics/cookies.rst new file mode 100644 index 000000000..ab5f3dca0 --- /dev/null +++ b/docs/topics/cookies.rst @@ -0,0 +1,138 @@ +.. _cookies: +.. _cookies-mw: + +======= +Cookies +======= + +Scrapy keeps track of the cookies that websites set and sends them back on +later requests to those websites, just like a web browser does. That is the job +of :class:`~scrapy.downloadermiddlewares.cookies.CookiesMiddleware`, which is +enabled by default. + + +Setting cookies on a request +============================ + +.. invisible-code-block: python + + from scrapy import Request + +Use the ``cookies`` parameter of :class:`~scrapy.Request` to send cookies of +your own, either as a dict: + +.. code-block:: python + + request = Request( + url="https://example.com", + cookies={"currency": "USD", "country": "UY"}, + ) + +Or as a list of dicts, which also lets you set cookie attributes: + +.. code-block:: python + + request = Request( + url="https://example.com", + cookies=[ + { + "name": "currency", + "value": "USD", + "domain": "example.com", + "path": "/currency", + "secure": True, + }, + ], + ) + +Setting attributes is only useful if the cookies are stored for later requests, +i.e. if :reqmeta:`dont_merge_cookies` is not enabled. + +.. caution:: Cookies set through the ``Cookie`` header are not handled by + :class:`~scrapy.downloadermiddlewares.cookies.CookiesMiddleware`, which + drops that header. + +.. caution:: When a cookie name or value is a byte sequence that is not UTF-8 + encoded, the cookie is dropped and a warning is logged. See + :ref:`topics-logging-advanced-customization` to customize the logging + behavior. + + +.. reqmeta:: cookiejar + +Multiple cookie sessions per spider +=================================== + +By default all requests share a single cookie jar (session). To use different +ones, pass an identifier in the :reqmeta:`cookiejar` request meta key: + +.. skip: next +.. code-block:: python + + for i, url in enumerate(urls): + yield Request(url, meta={"cookiejar": i}, callback=self.parse_page) + +The :reqmeta:`cookiejar` meta key is not "sticky", so you need to keep passing +it along on subsequent requests: + +.. code-block:: python + + def parse_page(self, response): + return Request( + "https://example.com/otherpage", + meta={"cookiejar": response.meta["cookiejar"]}, + callback=self.parse_other_page, + ) + + +.. reqmeta:: dont_merge_cookies + +Skipping the cookie jar for a request +===================================== + +Set the :reqmeta:`dont_merge_cookies` request meta key to ``True`` to keep a +request from touching the cookie jar in either direction: no stored cookie is +sent with the request, and no cookie received in the response is stored. The +cookies of the request itself are ignored as well. + + +.. setting:: COOKIES_ENABLED + +COOKIES_ENABLED +=============== + +Default: ``True`` + +Whether to enable :class:`~scrapy.downloadermiddlewares.cookies.CookiesMiddleware`. +If disabled, no cookies are sent to web servers. + + +.. setting:: COOKIES_DEBUG + +COOKIES_DEBUG +============= + +Default: ``False`` + +If enabled, Scrapy logs all cookies sent in requests (i.e. the ``Cookie`` +header) and all cookies received in responses (i.e. the ``Set-Cookie`` +header):: + + 2011-04-06 14:35:10-0300 [scrapy.core.engine] INFO: Spider opened + 2011-04-06 14:35:10-0300 [scrapy.downloadermiddlewares.cookies] DEBUG: Sending cookies to: + Cookie: clientlanguage_nl=en_EN + 2011-04-06 14:35:14-0300 [scrapy.downloadermiddlewares.cookies] DEBUG: Received cookies from: <200 http://www.diningcity.com/netherlands/index.html> + Set-Cookie: JSESSIONID=B~FA4DC0C496C8762AE4F1A620EAB34F38; Path=/ + Set-Cookie: ip_isocode=US + Set-Cookie: clientlanguage_nl=en_EN; Expires=Thu, 07-Apr-2011 21:21:34 GMT; Path=/ + 2011-04-06 14:49:50-0300 [scrapy.core.engine] DEBUG: Crawled (200) (referer: None) + [...] + + +CookiesMiddleware +================= + +.. module:: scrapy.downloadermiddlewares.cookies + :synopsis: Cookies Downloader Middleware + +.. autoclass:: CookiesMiddleware diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst index 66a0164d9..2d6ba0cf3 100644 --- a/docs/topics/downloader-middleware.rst +++ b/docs/topics/downloader-middleware.rst @@ -169,106 +169,10 @@ middleware, see the :ref:`downloader middleware usage guide For a list of the components enabled by default (and their orders) see the :setting:`DOWNLOADER_MIDDLEWARES_BASE` setting. -.. _cookies-mw: - CookiesMiddleware ----------------- -.. module:: scrapy.downloadermiddlewares.cookies - :synopsis: Cookies Downloader Middleware - -.. class:: CookiesMiddleware - - This middleware enables working with sites that require cookies, such as - those that use sessions. It keeps track of cookies sent by web servers, and - sends them back on subsequent requests (from that spider), just like web - browsers do. - - .. caution:: When non-UTF8 encoded byte sequences are passed to a - :class:`~scrapy.Request`, the ``CookiesMiddleware`` will log - a warning. Refer to :ref:`topics-logging-advanced-customization` - to customize the logging behaviour. - - .. caution:: Cookies set via the ``Cookie`` header are not considered by the - :ref:`cookies-mw`. If you need to set cookies for a request, use the - :class:`Request.cookies ` parameter. This is a known - current limitation that is being worked on. - -The following settings can be used to configure the cookie middleware: - -* :setting:`COOKIES_ENABLED` -* :setting:`COOKIES_DEBUG` - -.. reqmeta:: cookiejar - -Multiple cookie sessions per spider -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -There is support for keeping multiple cookie sessions per spider by using the -:reqmeta:`cookiejar` Request meta key. By default it uses a single cookie jar -(session), but you can pass an identifier to use different ones. - -For example: - -.. skip: next -.. code-block:: python - - for i, url in enumerate(urls): - yield scrapy.Request(url, meta={"cookiejar": i}, callback=self.parse_page) - -Keep in mind that the :reqmeta:`cookiejar` meta key is not "sticky". You need to keep -passing it along on subsequent requests. For example: - -.. code-block:: python - - def parse_page(self, response): - # do some processing - return scrapy.Request( - "http://www.example.com/otherpage", - meta={"cookiejar": response.meta["cookiejar"]}, - callback=self.parse_other_page, - ) - -.. setting:: COOKIES_ENABLED - -COOKIES_ENABLED -~~~~~~~~~~~~~~~ - -Default: ``True`` - -Whether to enable the cookies middleware. If disabled, no cookies will be sent -to web servers. - -Notice that despite the value of :setting:`COOKIES_ENABLED` setting if -``Request.``:reqmeta:`meta['dont_merge_cookies'] ` -evaluates to ``True`` the request cookies will **not** be sent to the -web server and received cookies in :class:`~scrapy.http.Response` will -**not** be merged with the existing cookies. - -For more detailed information see the ``cookies`` parameter in -:class:`~scrapy.Request`. - -.. setting:: COOKIES_DEBUG - -COOKIES_DEBUG -~~~~~~~~~~~~~ - -Default: ``False`` - -If enabled, Scrapy will log all cookies sent in requests (i.e. ``Cookie`` -header) and all cookies received in responses (i.e. ``Set-Cookie`` header). - -Here's an example of a log with :setting:`COOKIES_DEBUG` enabled:: - - 2011-04-06 14:35:10-0300 [scrapy.core.engine] INFO: Spider opened - 2011-04-06 14:35:10-0300 [scrapy.downloadermiddlewares.cookies] DEBUG: Sending cookies to: - Cookie: clientlanguage_nl=en_EN - 2011-04-06 14:35:14-0300 [scrapy.downloadermiddlewares.cookies] DEBUG: Received cookies from: <200 http://www.diningcity.com/netherlands/index.html> - Set-Cookie: JSESSIONID=B~FA4DC0C496C8762AE4F1A620EAB34F38; Path=/ - Set-Cookie: ip_isocode=US - Set-Cookie: clientlanguage_nl=en_EN; Expires=Thu, 07-Apr-2011 21:21:34 GMT; Path=/ - 2011-04-06 14:49:50-0300 [scrapy.core.engine] DEBUG: Crawled (200) (referer: None) - [...] +See :ref:`cookies`. DefaultHeadersMiddleware diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rst index a177d1ad5..1d97e39b6 100644 --- a/docs/topics/request-response.rst +++ b/docs/topics/request-response.rst @@ -53,65 +53,13 @@ Request objects ``None`` is passed as value, the HTTP header will not be sent at all. .. caution:: Cookies set via the ``Cookie`` header are not considered by the - :ref:`cookies-mw`. If you need to set cookies for a request, use the - ``cookies`` argument. This is a known current limitation that is being - worked on. + :ref:`cookie middleware `. If you need to set cookies for a + request, use the ``cookies`` argument. :type headers: dict - :param cookies: the request cookies. These can be sent in two forms. - - .. invisible-code-block: python - - from scrapy import Request - - 1. Using a dict: - - .. code-block:: python - - request_with_cookies = Request( - url="http://www.example.com", - cookies={"currency": "USD", "country": "UY"}, - ) - - 2. Using a list of dicts: - - .. code-block:: python - - request_with_cookies = Request( - url="https://www.example.com", - cookies=[ - { - "name": "currency", - "value": "USD", - "domain": "example.com", - "path": "/currency", - "secure": True, - }, - ], - ) - - The latter form allows for customizing the ``domain`` and ``path`` - attributes of the cookie. This is only useful if the cookies are saved - for later requests. - - .. reqmeta:: dont_merge_cookies - - When some site returns cookies (in a response) those are stored in the - cookies for that domain and will be sent again in future requests. - That's the typical behaviour of any regular web browser. - - Note that setting the :reqmeta:`dont_merge_cookies` key to ``True`` in - :attr:`request.meta ` causes custom cookies to be - ignored. - - For more info see :ref:`cookies-mw`. - - .. caution:: Cookies set via the ``Cookie`` header are not considered by the - :ref:`cookies-mw`. If you need to set cookies for a request, use the - :class:`scrapy.Request.cookies ` parameter. This is a known - current limitation that is being worked on. - + :param cookies: the request cookies, as a dict of cookie names and values + or as a list of dicts with a cookie each. See :ref:`cookies`. :type cookies: dict or list :param encoding: the encoding of this request (defaults to ``'utf-8'``). diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst index 0ea60e9cf..de665f42e 100644 --- a/docs/topics/settings.rst +++ b/docs/topics/settings.rst @@ -654,9 +654,8 @@ The default headers used for Scrapy HTTP Requests. They're populated in the :class:`~scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware`. .. caution:: Cookies set via the ``Cookie`` header are not considered by the - :ref:`cookies-mw`. If you need to set cookies for a request, use the - :class:`Request.cookies ` parameter. This is a known - current limitation that is being worked on. + :ref:`cookie middleware `. If you need to set cookies for a + request, use the :class:`Request.cookies ` parameter. .. caution:: A ``Referer`` header defined here only reaches requests for which :class:`~scrapy.spidermiddlewares.referer.RefererMiddleware` does not set