diff --git a/docs/index.rst b/docs/index.rst index 779647030..898f33249 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -134,6 +134,7 @@ Solving specific problems topics/images topics/ubuntu topics/scrapyd + topics/jobs :doc:`faq` Get answers to most frequently asked questions. @@ -156,6 +157,9 @@ Solving specific problems :doc:`topics/scrapyd` Deploying your Scrapy project in production. +:doc:`topics/jobs` + Learn how to pause and resume crawls for large spiders. + .. _extending-scrapy: Extending Scrapy diff --git a/docs/topics/jobs.rst b/docs/topics/jobs.rst new file mode 100644 index 000000000..c2b9f104b --- /dev/null +++ b/docs/topics/jobs.rst @@ -0,0 +1,99 @@ +.. _topics-jobs: + +================================= +Jobs: pausing and resuming crawls +================================= + +Sometimes, for big sites, it's desirable to pause crawls and be able to resume +them later. + +Scrapy supports this functionality out of the box by providing the following +facilities: + +* a scheduler that persists scheduled requests on disk + +* a duplicates filter that persists visited requests on disk + +* an extension that keeps some spider state (key/value pairs) persistent + between batches + +Job directory +============= + +To enable persistence support you just need to define a *job directory* through +the ``JOBDIR`` setting. This directory will be for storing all required data to +keep the state of a single job (ie. a spider run). It's important to note that +this directory must not be shared by different spiders, or even different +jobs/runs of the same spider, as it's meant to be used for storing the state of +a *single* job. + +How to use it +============= + +To start a spider with persistence supported enabled, run it like this:: + + scrapy crawl somespider -s JOBDIR=crawls/somespider-1 + +Then, you can stop the spider safely at any time (by pressing Ctrl-C or sending +a signal), and resume it later by issuing the same command:: + + scrapy crawl somespider -s JOBDIR=crawls/somespider-1 + +Keeping persitent state between batches +======================================= + +Sometimes you'll want to keep some persistent spider state between pause/resume +batches. You can use the ``spider.state`` attribute for that, which should be a +dict. There's a built-in extension that takes care of serializing, storing and +loading that attribute from the job directory, when the spider starts and +stops. + +Here's an example of a callback that uses the spider state (other spider code +is ommited for brevity):: + + def parse_item(self, response): + # parse item here + self.state['items_count'] = self.state.get('items_count', 0) + 1 + +Persistence gotchas +=================== + +There are a few things to keep in mind if you want to be able to use the Scrapy +persistence support: + +Cookies expiration +------------------ + +Cookies may expire. So, if you don't resume your spider quickly the requests +scheduled may no longer work. This won't be an issue if you spider doesn't rely +on cookies. + +Request serialization +--------------------- + +Requests must be serializable by the `pickle` module, in order for persistence +to work, so you should make sure that your requests are serializable. + +The most common issue here is to use ``lambda`` functions on request callbacks that +can't be persisted. + +So, for example, this won't work:: + + def some_callback(self, response): + somearg = 'test' + return Request('http://www.example.com', callback=lambda r: self.other_callback(r, somearg)) + + def other_callback(self, response, somearg): + print "the argument passed is:", somearg + +But this will:: + + def some_callback(self, response): + somearg = 'test' + return Request('http://www.example.com', meta={'somearg': somearg}) + + def other_callback(self, response): + somearg = response.meta['somearg'] + print "the argument passed is:", somearg + +.. _pickle: http://docs.python.org/library/pickle.html