RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
9,472 Commits 49 Branches 132 Tags 182 MiB
Commit Graph

47 Commits

Author SHA1 Message Date
Emmanuel Rondan e211ec0aa2 adding black formatter to all the code 2022-11-29 11:30:46 -03:00
Matthew Donoughe 40d9ca3bdd
use pathlib 2022-10-17 17:40:10 -04:00
Mikhail Korobov 1054689593 remove unused imports 
thanks flake8!
 2022-07-18 23:53:30 +05:00
Mikhail Korobov e248360e6e remove compatibility code from tests for the case dataclasses module is not available 
It was Python 3.6 compat code, and Python 3.6 support is dropped.
 2022-07-18 23:49:08 +05:00
Adrián Chaves c5627af15b
Centralize request fingerprints (#4524) 
Co-authored-by: Mikhail Korobov <kmike84@gmail.com>
 2022-06-07 18:44:54 +02:00
Maxime Nannan 078622cfb0
Fix file expiration issue with GCS (#5318) 2022-05-20 08:30:06 +02:00
Adrián Chaves 6ef7c44061 Fix timezone test issue 2020-09-22 12:45:21 +02:00
Adrián Chaves 8f46e84519 Fix style issues 2020-09-21 23:28:16 +02:00
Adrián Chaves c3b740f078 Use mocking for tests/test_pipeline_files.py::TestS3FilesStore::test_persist 2020-09-21 23:25:37 +02:00
Andrey Rahmatullin 5e997587d9
Remove dead boto2 code, deprecate is_botocore() (#4776) 2020-09-20 15:06:46 +02:00
Ammar Najjar 58ca8bbf6d Use f-strings (#4307) 2020-08-26 07:11:05 +02:00
Ajay Mittur 1c4b4cc6b0
Support defining file path based on item in media pipelines (#4686) 2020-08-11 14:12:44 +02:00
Eugenio Lacuesta ec98dabfab
Support for dataclass and attrs items (#3881) 2020-06-14 11:45:27 +02:00
Mikhail Korobov febe82a907
Merge pull request #4486 from ilias-ant/add-file-status-on-media-pipelines-file-info 
Add status (downloaded, uptodate) to files information
 2020-05-18 22:30:30 +05:00
ilias-ant 773ddf77ba added more tests to cover the (downloaded, uptodate, cached) status functionality 2020-04-19 14:14:17 +03:00
Eugenio Lacuesta 182394bcec
E501 compliance (tests) 2020-03-22 23:29:30 -03:00
Marc Hernandez Cabot b49ece0b87 fix E701 and E271 flake8 2020-02-22 12:32:38 +01:00
Marc Hernandez Cabot 6fb85951ce fix E22X flake8 2020-02-21 08:39:14 +01:00
Marc Hernández 91bbc70bc1
fix E30X flake8 (#4355) 2020-02-21 06:05:31 +01:00
OmarFarrag 40e0a11aa8 Fix Flake8 errors 2020-01-24 15:51:48 +02:00
OmarFarrag fc98aa6b67
Merge branch 'master' into ftp_files#3928 2020-01-24 14:52:40 +02:00
Marc Hernandez Cabot a5de2c64e6 fix W291, W292, W293 (whitespaces) 2019-12-18 16:33:46 +01:00
Eugenio Lacuesta 00b793dc59
Remove elluding six occurrences 2019-11-03 12:26:38 -03:00
Eugenio Lacuesta 5ab0f189ce
Remove six.moves occurrences from tests 2019-11-03 00:01:09 -03:00
Andrey Rakhmatullin 75b1d051d9 Simplify some more imports. 2019-10-31 15:22:45 +05:00
OmarFarrag 13735bcf34 Disallow media extensions unregistered with IANA (#3954) 
Co-Authored-By: s-sanjay <sanjay537@gmail.com>
 2019-09-16 14:04:06 +02:00
OmarFarrag 0e8770a2f4 test for files pipeline ftp store 2019-09-06 15:47:57 +02:00
nyov 7a398b7086 Deprecation removals for Scrapy 1.7 
Removing deprecations of 2015 and prior (pre-1.1)
 2019-07-06 13:41:56 +00:00
rhoboro 5254ac393b added test for gcs policy 2018-04-03 18:06:34 +09:00
rhoboro e5d4364b2a Add tests for GCS Storage 2017-09-13 16:24:16 +09:00
Paul Tremberth 07f9985a94 TST: Randomize FILES_EXPIRES above 90 days 2016-12-21 17:03:11 +01:00
pawelmhm 7d88209543 [image & file pipeline] loading setting for user classes 
if user has some custom subclass of Image pipeline and no setting for
this pipeline, he should get default settings defined for Image Pipeline.

Fixes #2198
 2016-09-15 09:39:16 +02:00
Pawel Miech fa4d0cdfe5 [FilesPipeline, ImagesPipeline] fix for cls attrs with DEFAULT prefix 
some class attributes for ImagePipeline and FilesPipeline had DEFAULT prefix. These
attributes should be preserved as well, if users subclasses define values for
DEFAULT_<CLS_ATTRIBUTE_NAME> attribute this value should be preserved.
 2016-06-20 12:53:20 +02:00
Pawel Miech 539d34bce0 [media-pipeline, file-pipeline] allow setting custom settings for subclasses 
* move key_for_pipe function to media pipeline so that file pipeline can use it
* use key_for_pipe in file pipeline so that users can define custom settings for subclasses easily
* add tests for file pipelines attributes and settings
 2016-06-15 15:39:11 +02:00
Pawel Miech acbfdc6184 [files_pipeline] ensure class attributes are preserved 
dont override class attributes with default settings (same as in image pipeline).
 2016-06-15 15:14:28 +02:00
Pawel Miech c6d1686d98 [files_pipeline] unify tests for files pipeline 
if test tests same thing but for different field it can be unified into one.
 2016-06-15 14:48:25 +02:00
djunzu 8228a0c491 Change FilesPipeline class attributes to instance attributes. 
modified:   scrapy/pipelines/files.py
	modified:   tests/test_pipeline_files.py
 2016-03-31 19:20:39 -03:00
djunzu e9d48f8a8e Add tests. 
modified:   tests/test_pipeline_files.py
	modified:   tests/test_pipeline_images.py
 2016-03-31 19:19:49 -03:00
Konstantin Lopuhin d61fbcc8b5 Support headers in S3FilesStore.persist_file for botocore 2016-02-18 10:57:02 +03:00
Konstantin Lopuhin 3cb7a567ea py3 fix for TestS3FilesStore: checksum is a native string 2016-02-15 19:59:48 +03:00
Konstantin Lopuhin 32cd8c9165 add direct test for S3FilesStore 2016-02-15 19:59:48 +03:00
nyov 991197003b PY3 fix tests pipelines files 2015-08-08 04:22:58 +05:00
nyov ec8afbc060 PY3 fix test pipeline media 2015-07-29 13:51:01 +00:00
Julia Medina 8021df18d4 Move scrapy/contrib/pipeline to scrapy/pipelines 2015-04-29 21:26:35 -03:00
Mikhail Korobov 39635e5f55 Allow spiders to return dicts. See GH-1064. 2015-03-18 07:26:56 +05:00
Berker Peksag 31e5f164d4 Import unittest.mock if available. 
mock is in the stdlib since Python 3.3.
 2015-03-08 15:03:04 +02:00
Daniel Graña 242c08553f move tests under root dir 2014-07-30 19:45:08 -03:00