Fix a race condition in the FilesPipeline

Checksum calculation could happen simultaniously with
persisting the file in the store (which is done in a thread):
they operated on the same buf object.
Concretely this lead to a bug with S3FilesStore
when using botocore: the signature did not match because
the position in the buf was already at the end.
The fix is to move checksum calculation before passing buf
to the store.
This commit is contained in:
Konstantin Lopuhin 2016-03-27 21:29:21 +02:00
parent 5045a4f168
commit fc8cd45a48
1 changed files with 2 additions and 1 deletions

View File

@ -364,8 +364,9 @@ class FilesPipeline(MediaPipeline):
def file_downloaded(self, response, request, info):
path = self.file_path(request, response=response, info=info)
buf = BytesIO(response.body)
self.store.persist_file(path, buf, info)
checksum = md5sum(buf)
buf.seek(0)
self.store.persist_file(path, buf, info)
return checksum
def item_completed(self, results, item, info):