RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,393 Commits 49 Branches 132 Tags 155 MiB
Commit Graph

189 Commits

Author SHA1 Message Date
joaquin garmendia 85604e1078
Add failed and success count stats to feedstorage backends (#4850) 2020-11-11 21:16:01 +01:00
Andrey Rahmatullin 1aeda66435
Merge pull request #4804 from Gallaecio/mock-s3 
Use mocking for S3 tests that currently need server credentials
 2020-10-06 18:23:01 +05:00
Mikhail Korobov 4f27c5f82b
Merge pull request #4768 from maranqz/feature/4606-exporter-from-FEEDS 
Pass info from FEEDS to ItemExporter
 2020-10-02 00:10:05 +05:00
Adrián Chaves 8f46e84519 Fix style issues 2020-09-21 23:28:16 +02:00
Adrián Chaves 35726da434 tests/test_feedexport.py: remove unused import 2020-09-21 22:55:25 +02:00
Adrián Chaves 17e135377a Use mocking for tests/test_feedexport.py::BatchDeliveriesTest::test_s3_export 2020-09-21 22:54:39 +02:00
Adrián Chaves 56f05fb164 Use mocking for tests/test_feedexport.py::S3FeedStorageTest::test_store 2020-09-21 22:01:09 +02:00
Andrey Rahmatullin 5e997587d9
Remove dead boto2 code, deprecate is_botocore() (#4776) 2020-09-20 15:06:46 +02:00
maranqz 71d2c2f1a3 improve view of dict 2020-08-30 12:43:44 +03:00
maranqz a8e895e684 kwargs for Item exporters classes 
test
docs
 2020-08-30 10:57:22 +03:00
Ammar Najjar 7597193dbe
Merge branch 'master' into 4307-use-f-strings 
Resolve Conflicts:
	tests/test_middleware.py
 2020-08-26 13:46:46 +02:00
nyov cf50561b86
Allow passing classes directly in Settings (#3873) 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-08-26 13:08:14 +02:00
Ammar Najjar 58ca8bbf6d Use f-strings (#4307) 2020-08-26 07:11:05 +02:00
Adrián Chaves e70975f0bb
Allow overwriting feeds (#4512) 
Co-authored-by: Yuval Hager <yhager@yhager.com>
 2020-08-17 15:10:08 +02:00
Mikhail Korobov 61459de476
Merge pull request #4688 from ashellunts/obsolete-s3feednokeys 
Remove obsolete S3FeedStorage instancing without AWS credentials
 2020-08-17 13:58:32 +05:00
Adrián Chaves 5138f9a965 Merge remote-tracking branch 'upstream/master' into allow-customizing-export-column-names 2020-08-11 13:42:05 +02:00
Artur Shellunts 5ec66be4a4
Merge branch 'master' into obsolete-s3feednokeys 2020-07-30 22:18:00 +02:00
Adrián Chaves 890b2138a6 Remove the u prefix from strings 2020-07-30 13:39:30 +02:00
Artur Shellunts 430d22e46e Remove not used import warnings 2020-07-21 23:39:04 +02:00
nyov 2829cd4268 Allow use without credentials 2020-07-21 23:38:02 +02:00
nyov 98e8086d1b Adapt S3FeedStorage testcase 2020-07-21 23:38:02 +02:00
nyov ece4fa6c7c Fix ignored testcase: boto is never installed 2020-07-21 23:38:01 +02:00
BroodingKangaroo 3e0492741d Another try to fix test errors on Windows 2020-07-19 00:10:29 +03:00
BroodingKangaroo 86f7ac2f2b Try to fix error at Windows 2020-07-17 18:22:47 +03:00
Adrián Chaves c4f9250220
Merge branch 'master' into ISSUE-4250-add_batch_deliveries 2020-07-16 17:44:18 +02:00
Mikhail Korobov 07470e1a3c
Merge pull request #3608 from ejulio/feat-685 
[MRG+1] Fix for #685 Add Google Cloud Storage Feed Export
 2020-07-16 18:11:19 +05:00
BroodingKangaroo 6454d456d2 Make check of placeholder less strict 2020-07-03 10:24:13 +03:00
Adrián Chaves 3199048520 Complete Azure Pipelines CI setup 2020-07-03 00:56:24 +02:00
BroodingKangaroo 1e245046ed Change setting name. Add leading zeroes to batch_id. Minor fixes. 2020-07-02 12:38:08 +03:00
BroodingKangaroo 7b1d3c35ea Minor updates 2020-07-01 12:24:31 +03:00
BroodingKangaroo 05c2587c6a Docs update and tiny fixes 2020-06-28 09:45:45 +03:00
BroodingKangaroo 88a52198b9 Add batch_item_count support in FEEDS setting 2020-06-27 12:45:08 +03:00
BroodingKangaroo c3cee74fd4 Change default value of FEED_STORAGE_BATCH_ITEM_COUNT to 0 2020-06-26 18:45:21 +03:00
BroodingKangaroo 77333666f1 resolve conflicts 2020-06-26 14:55:46 +03:00
Eugenio Lacuesta c7f1c7e3f3
Merge branch 'master' into feat-685 2020-06-19 21:30:54 -03:00
Stanislau Hluboki a4bfd5ab6f Fix duplicated feed logs 2020-06-19 17:51:56 +03:00
Adrián Chaves f9c63384c0
Merge branch 'master' into allow-customizing-export-column-names 2020-06-18 14:09:20 +02:00
Stas Glubokiy 3d027fb578
Fix missing storage.store calls in FeedExporter.close_spider (#4626) 2020-06-17 17:08:14 +02:00
BroodingKangaroo 677e619d37 Fix too long lines 2020-05-21 14:57:03 +03:00
BroodingKangaroo 5f6c1dceb1 Merge remote-tracking branch 'upstream/master' into ISSUE-4250-add_batch_deliveries 2020-05-21 12:43:04 +03:00
BroodingKangaroo 10ae1a284f Minor fixes 2020-05-15 22:50:54 +03:00
Eugenio Lacuesta 3eeecb4273
Merge branch 'master' into flake8-max-line-length 2020-05-15 11:11:20 -03:00
BroodingKangaroo 111a58fe3d Merge remote-tracking branch 'upstream/master' into ISSUE-4250-add_batch_deliveries 2020-05-14 09:53:14 +03:00
BroodingKangaroo 2327ecead0 Rename FEED_STORAGE_BATCH_SIZE to FEED_STORAGE_BATCH_ITEM_COUNT 2020-05-13 22:50:04 +03:00
Eugenio Lacuesta 6f8758624c
Flake8: remove F841 2020-05-11 13:50:34 -03:00
Adrián Chaves a859ea0c8b Merge remote-tracking branch 'upstream/master' into allow-customizing-export-column-names 2020-05-06 19:48:32 +02:00
BroodingKangaroo dad2ea7522 Change time_id to batch_time 2020-05-02 01:21:03 +03:00
BroodingKangaroo 3f9874fac9 Add test s3 export 2020-05-01 11:52:16 +03:00
BroodingKangaroo f0f1be76d1 Using time_id instead of time as a timestamp 2020-04-27 09:56:57 +03:00
BroodingKangaroo ec76445dd6 Update tests 2020-04-18 09:29:23 +03:00
Eugenio Lacuesta 7926da4bd0
Merge branch 'master' into flake8-max-line-length 2020-04-16 16:05:09 -03:00
BroodingKangaroo 5980ae72c6 Some minor fixes and refactoring 2020-04-16 10:13:39 +03:00
BroodingKangaroo 16836e9e77 Merge branch 'ISSUE-4250-add_batch_deliveries' of https://github.com/BroodingKangaroo/scrapy into ISSUE-4250-add_batch_deliveries 2020-04-16 00:41:21 +03:00
BroodingKangaroo 963580463b Update tests 2020-04-15 23:39:46 +03:00
BroodingKangaroo 39d0d13d3f Add partial deliveries tests 2020-04-15 23:38:27 +03:00
Eugenio Lacuesta 1bd8f392c9
Initial removal of twisted.internet.defer.returnValue 2020-04-13 06:12:30 -03:00
Eugenio Lacuesta 182394bcec
E501 compliance (tests) 2020-03-22 23:29:30 -03:00
Eugenio Lacuesta 34d607194b
Merge branch 'master' into feat-685 2020-03-18 21:24:36 -03:00
elacuesta dfbe1d9507
Remove object base class (#4430) 2020-03-16 20:12:46 +01:00
Eugenio Lacuesta ada37c5409
Export to multiple formats in a single crawl 2020-03-05 09:33:57 -03:00
Adrián Chaves 5834088e67 Apply feedback 2020-02-18 14:18:15 +01:00
Adrián Chaves e1be078eaa Fix Flake8-reported issues 2020-02-12 17:38:06 +01:00
Adrián Chaves 96bb3b5144
Merge branch 'master' into feat-685 2020-02-10 19:45:09 +01:00
Adrián Chaves ab13221b0b Merge remote-tracking branch 'upstream/master' into allow-customizing-export-column-names 2019-12-17 14:06:14 +01:00
Eugenio Lacuesta 5bab3c0261
Merge remote-tracking branch 'upstream/master' into remove-six-code 2019-11-22 12:12:29 -03:00
Adrián Chaves a2bf340bab Remove unused imports 2019-11-21 14:21:17 +01:00
Eugenio Lacuesta 05785c1c17
Merge remote-tracking branch 'upstream/master' into remove-six-code 2019-11-19 11:01:34 -03:00
Andrey Rakhmatullin a138fb05d4 Replace to_native_str calls with to_unicode. 2019-11-14 13:45:39 +05:00
Andrey Rakhmatullin 92ffd2f249 Simplify some more imports. 2019-11-14 13:45:39 +05:00
Andrey Rahmatullin c91390558d
Merge pull request #4074 from purvaudai/master 
Added Pathlib.Path support: Issue #3731
 2019-11-12 21:43:28 +05:00
purvaudai 4b8b0345e5 Mades Changes as per review 2019-11-12 18:17:15 +05:30
purvaudai 50eaabe1fc Added Test 2019-11-11 20:00:26 +05:30
purvaudai f39ff49458 Added Test 2019-11-11 18:54:21 +05:30
purvaudai 970c3be160 Added Test 2019-11-11 18:34:15 +05:30
purvaudai 9e6e2dde2b Adding test 2019-11-11 16:10:37 +05:30
purvaudai 0042c389eb Adding test 2019-11-11 15:57:58 +05:30
purvaudai 084a1cda6d Adding test 2019-11-11 15:41:00 +05:30
Eugenio Lacuesta 5ab0f189ce
Remove six.moves occurrences from tests 2019-11-03 00:01:09 -03:00
Eugenio Lacuesta c0bfaef37a
Remove __future__ imports from tests 2019-11-02 23:27:04 -03:00
Andrey Rakhmatullin 397e883556 Replace to_native_str calls with to_unicode. 2019-10-31 15:22:53 +05:00
Andrey Rakhmatullin 75b1d051d9 Simplify some more imports. 2019-10-31 15:22:45 +05:00
Adrián Chaves 1d5c270ce8
Fix dangling file descriptor in FeedExporter when FEED_STORE_EMPTY is False (#4023) 2019-10-22 15:12:52 +02:00
purvaudai 4184bac068 Added Pathlib.Path test 2019-10-22 16:57:14 +05:30
purvaudai 85f56a92f0 Added Pathlib.Path test 2019-10-22 16:43:17 +05:30
purvaudai 7031e3a124 Added Pathlib.Path test 2019-10-22 16:31:14 +05:30
purvaudai cd4c211f4b Added Pathlib.Path test 2019-10-22 15:38:06 +05:30
purvaudai a776554282 Added Pathlib.Path test 2019-10-22 15:31:55 +05:30
purvaudai ad96d6ef59 Added Pathlib.Path test correctly 2019-10-22 14:53:59 +05:30
purvaudai 2ee38e8ddb Added Pathlib.Path test 2019-10-22 14:43:47 +05:30
Júlio César Batista b84f99ff5d Merge 2019-08-29 11:11:56 -03:00
Júlio César Batista 110bc92e6b Fix default value of FEED_STORAGE_GCS_ACL 2019-08-29 11:10:00 -03:00
Pengyu Chen 7b755a41a1
Added: Properly handling quoted passwords in FEED_URI for FTP 2019-08-06 15:19:57 +01:00
Luiz Francisco Rodrigues da Silva df68c4b9b1 Add a setting to use active mode in FTPFeedStorage (#3829) 
Add a setting to use active mode in FTPFeedStorage

FTP servers can be configured in active and passive mode, by default
Python and Scrapy use passive mode and there was no way to use active.

This commit adds a setting FEED_STORAGE_FTP_ACTIVE to allow a feed
to be exported to a FTP server configured in active mode.
 2019-07-10 14:25:52 +02:00
nyov 5442c2d3c3 Updating S3FeedStorage instancing without AWS key. 2019-07-06 13:41:56 +00:00
Adrián Chaves 484bd0d22a Allow customizing export column names 2019-03-29 15:16:54 +01:00
Adrián Chaves ba13de29e1
Merge branch 'master' into feat-685 2019-03-26 10:38:38 +01:00
Victor Torres fda1d04b81
Merge branch 'master' into feed-storage-s3-acl 2019-03-15 13:26:45 -03:00
Adrián Chaves 858f5be747 backwards → backward (adj.) 2019-03-01 16:10:23 +01:00
Júlio César Batista 0bb3d8ca93 Updating Google Cloud Storage scheme to gs instead of gcs 2019-02-27 18:41:01 -03:00
Victor Torres 9b8ba4c383 try to import botocore before runing some tests 2019-02-14 16:20:56 -02:00