5.8 KiB
Sessions
A session is the state that a group of requests share, starting with their cookies, which Scrapy stores and sends back on later requests to the same website, like a web browser does.
Every request uses the same session unless told otherwise, so a crawl behaves like a single browser profile. Use more than one to keep parts of a crawl from sharing state, e.g. to crawl a website through several independent profiles at the same time.
Choosing the session of a request
System Message: ERROR/3 (<stdin>, line 24)
Unknown directive type "versionadded".
.. versionadded:: VERSION
System Message: ERROR/3 (<stdin>, line 30)
Unknown directive type "reqmeta".
.. reqmeta:: session
Set the :reqmeta:`session` request meta key to a session ID:
System Message: ERROR/3 (<stdin>, line 32); backlink
Unknown interpreted text role "reqmeta".System Message: WARNING/2 (<stdin>, line 35)
Cannot analyze code. Pygments package not found.
.. code-block:: python
for index, url in enumerate(urls):
yield Request(url, meta={"session": index})
Any ID works, and the session is created the first time it is used. :meth:`~scrapy.sessions.Sessions.create` covers the case where you have no ID of your own to give: it returns a session that is certainly new, which indexing cannot promise, since the ID you choose may be in use already.
System Message: ERROR/3 (<stdin>, line 40); backlink
Unknown interpreted text role "meth".System Message: WARNING/2 (<stdin>, line 46)
Cannot analyze code. Pygments package not found.
.. code-block:: python
session = self.crawler.sessions.create()
yield Request(url, meta={"session": session.id})
The follow-up requests that a spider callback yields stay in the session of the request that got that callback its response, so a session lasts as long as the crawl that follows from it without its ID being passed along by hand. Set :reqmeta:`session` on one of those requests to move it to a different session.
System Message: ERROR/3 (<stdin>, line 51); backlink
Unknown interpreted text role "reqmeta".A request that neither sets :reqmeta:`session` nor inherits one uses the session whose ID is "main". Everything a crawl does without asking for a session happens there, which makes session="main" the way to send a request back to it, e.g. from a callback whose own session is a different one.
System Message: ERROR/3 (<stdin>, line 56); backlink
Unknown interpreted text role "reqmeta".Set :reqmeta:`session` to None for a request to use no session at all: no stored cookie is sent with it and no cookie received in its response is stored. The cookies of the request itself are still sent, and its follow-up requests inherit the lack of a session.
System Message: ERROR/3 (<stdin>, line 61); backlink
Unknown interpreted text role "reqmeta".The session registry
System Message: ERROR/3 (<stdin>, line 72)
Unknown directive type "versionadded".
.. versionadded:: VERSION
Sessions live in :attr:`Crawler.sessions <scrapy.crawler.Crawler.sessions>`, where you can inspect and modify them:
System Message: ERROR/3 (<stdin>, line 74); backlink
Unknown interpreted text role "attr".System Message: WARNING/2 (<stdin>, line 78)
Cannot analyze code. Pygments package not found.
.. code-block:: pycon
>>> cookies = crawler.sessions["main"].cookies
>>> len(cookies)
2
>>> cookies.clear()
When a session stops working, e.g. because the website expired it, call :meth:`~scrapy.sessions.Sessions.retire` and retry the request with :func:`~scrapy.downloadermiddlewares.retry.get_retry_request`: the session is gone, so the retry starts a new one under the same ID.
System Message: ERROR/3 (<stdin>, line 87); backlink
Unknown interpreted text role "meth".System Message: ERROR/3 (<stdin>, line 87); backlink
Unknown interpreted text role "func".System Message: ERROR/3 (<stdin>, line 92)
Unknown directive type "autoclass".
.. autoclass:: scrapy.sessions.Sessions
:members:
System Message: ERROR/3 (<stdin>, line 95)
Unknown directive type "autoclass".
.. autoclass:: scrapy.sessions.Session
:members:
SESSIONS_MAX
Default: 1000
Maximum number of sessions to keep in memory. When the limit is reached, the session that has not been used for the longest time is dropped, losing its cookies. The first drop is logged as a warning, and :stat:`sessions/dropped` counts them all.
System Message: ERROR/3 (<stdin>, line 149); backlink
Unknown interpreted text role "stat".Raise it if a crawl needs more sessions alive at the same time, e.g. because it gives every request a session of its own.
System Message: ERROR/3 (<stdin>, line 158)
Unknown directive type "setting".
.. setting:: COOKIES_ENABLED