mirror of https://github.com/scrapy/scrapy.git
Fix a race condition in the FilesPipeline
Checksum calculation could happen simultaniously with persisting the file in the store (which is done in a thread): they operated on the same buf object. Concretely this lead to a bug with S3FilesStore when using botocore: the signature did not match because the position in the buf was already at the end. The fix is to move checksum calculation before passing buf to the store.
This commit is contained in:
parent
5045a4f168
commit
fc8cd45a48
|
|
@ -364,8 +364,9 @@ class FilesPipeline(MediaPipeline):
|
|||
def file_downloaded(self, response, request, info):
|
||||
path = self.file_path(request, response=response, info=info)
|
||||
buf = BytesIO(response.body)
|
||||
self.store.persist_file(path, buf, info)
|
||||
checksum = md5sum(buf)
|
||||
buf.seek(0)
|
||||
self.store.persist_file(path, buf, info)
|
||||
return checksum
|
||||
|
||||
def item_completed(self, results, item, info):
|
||||
|
|
|
|||
Loading…
Reference in New Issue