mirror of https://github.com/scrapy/scrapy.git
Fix a race condition in the FilesPipeline
Checksum calculation could happen simultaniously with persisting the file in the store (which is done in a thread): they operated on the same buf object. Concretely this lead to a bug with S3FilesStore when using botocore: the signature did not match because the position in the buf was already at the end. The fix is to move checksum calculation before passing buf to the store.
This commit is contained in:
parent
84371ffb22
commit
b01bf4728f
|
|
@ -364,8 +364,9 @@ class FilesPipeline(MediaPipeline):
|
|||
def file_downloaded(self, response, request, info):
|
||||
path = self.file_path(request, response=response, info=info)
|
||||
buf = BytesIO(response.body)
|
||||
self.store.persist_file(path, buf, info)
|
||||
checksum = md5sum(buf)
|
||||
buf.seek(0)
|
||||
self.store.persist_file(path, buf, info)
|
||||
return checksum
|
||||
|
||||
def item_completed(self, results, item, info):
|
||||
|
|
|
|||
Loading…
Reference in New Issue