Fix a race condition in the FilesPipeline

Checksum calculation could happen simultaniously with
persisting the file in the store (which is done in a thread):
they operated on the same buf object.
Concretely this lead to a bug with S3FilesStore
when using botocore: the signature did not match because
the position in the buf was already at the end.
The fix is to move checksum calculation before passing buf
to the store.
This commit is contained in:
Konstantin Lopuhin 2016-03-27 21:29:21 +02:00 committed by Paul Tremberth
parent 84371ffb22
commit b01bf4728f
1 changed files with 2 additions and 1 deletions

View File

@ -364,8 +364,9 @@ class FilesPipeline(MediaPipeline):
def file_downloaded(self, response, request, info):
path = self.file_path(request, response=response, info=info)
buf = BytesIO(response.body)
self.store.persist_file(path, buf, info)
checksum = md5sum(buf)
buf.seek(0)
self.store.persist_file(path, buf, info)
return checksum
def item_completed(self, results, item, info):