RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
9,472 Commits 49 Branches 132 Tags 132 MiB
Commit Graph

160 Commits

Author SHA1 Message Date
Emmanuel Rondan e211ec0aa2 adding black formatter to all the code 2022-11-29 11:30:46 -03:00
Matthew Donoughe 7daf735f45
downgrade type hint syntax 2022-10-18 20:07:29 -04:00
Matthew Donoughe 40d9ca3bdd
use pathlib 2022-10-17 17:40:10 -04:00
Laerte Pereira 3ca7877781
chore: Skip `batch_path_differ` test on Windows 2022-09-29 11:51:50 -03:00
Mikhail Korobov b103664bf4
Address 2/3 of warnings from tests (#5561) 2022-07-19 17:39:26 +02:00
Adrián Chaves ec5cf3e9ce test_feedexport: solve ordered comparison issues 2022-06-17 09:10:18 +02:00
Adrián Chaves bc285f393c Revert "Revert "test_feedexport: remove ordered=False"" 
This reverts commit 516e2d6ec0.
 2022-06-17 09:00:39 +02:00
Adrián Chaves 516e2d6ec0 Revert "test_feedexport: remove ordered=False" 
This reverts commit 24f382fa45.
 2022-06-17 08:55:45 +02:00
Adrián Chaves 3729c6d266 Remove unused import and redundant import 2022-06-17 08:33:34 +02:00
Adrián Chaves 24f382fa45 test_feedexport: remove ordered=False 2022-06-17 08:31:45 +02:00
Adrián Chaves 1b9ed22bec Remove Python < 3.7 leftover 2022-06-17 08:27:17 +02:00
Adrián Chaves 3e994bda45 Update for Python 3.7+ 2022-06-16 20:53:14 +02:00
Adrián Chaves e8503217fb Merge remote-tracking branch 'upstream/master' into allow-customizing-export-column-names 2022-06-16 20:19:52 +02:00
Sixuan (Cherie) Wu 30d5779ea9
Fix FEED_URI_PARAMS: custom params throws KeyError (#4966) 
* fix FEED_URI_PARAMS: custom params throws KeyError closes #4962

* another try FEED_URI_PARAMS

* add warning message and change default function

* Add tests for FEED_URI_PARAMS

* FEED_URI_PARAMS: warn if the params dict has been modified in-place

* [Doc] FEED_URI_PARAMS: modifying params in-place is deprecated

* Remove whileline

* Rename parameters for lambda function

* Type hints for FeedExporter._get_uri_params

Co-authored-by: Adrián Chaves <adrian@chaves.io>
Co-authored-by: Eugenio Lacuesta <eugenio.lacuesta@gmail.com>
 2022-01-28 14:30:30 -03:00
D R Siddhartha 43ea21e830
Feed post-processing plugin support (#5190) 2021-08-24 11:48:01 +02:00
Mannan2812 94baa4b272
Fix FileFeedStoragePreFeedOptionsTest fails in CI/CD pipeline (#5198) 2021-08-05 21:23:11 +02:00
laggardkernel 7e4321f201 Add support for temporary security credential in AWS auth 2021-07-27 19:39:59 +08:00
Türkalp Burak KAYRANCIOĞLU fcc6becc58
S3FeedStorage: allow custom endpoint (#4998) 
Co-authored-by: Andrey Rahmatullin <wrar@wrar.name>
 2021-07-14 10:00:43 +02:00
D R Siddhartha 4ddc9d6b55
Feeds: Item Filters (#5178) 2021-07-13 17:22:29 +02:00
Vostretsov Nikita 23cfdb058e
Reducing amount of warnings during test run (#5162) 
* put flake8 options into separate file to remove pytest warnings

* remove ResourceLeaked warning in pypy

* suppress warnings from twisted

* ignore deprecation warnings here

* ignore deprecation warning in tests of deprecated methods

* ignore deprecation warnings here

* update test classes

* don`t use deprecated method call

* ignore deprecation warnings here

* proper warning class

* more selective ignoring

* Revert "don`t use deprecated method call"

This reverts commit 59216ab560.
 2021-05-28 14:45:06 +05:00
Adrián Chaves 07379cf9b7
Merge branch 'master' into allow-customizing-export-column-names 2020-11-20 20:06:18 +01:00
joaquin garmendia 85604e1078
Add failed and success count stats to feedstorage backends (#4850) 2020-11-11 21:16:01 +01:00
Andrey Rahmatullin 1aeda66435
Merge pull request #4804 from Gallaecio/mock-s3 
Use mocking for S3 tests that currently need server credentials
 2020-10-06 18:23:01 +05:00
Mikhail Korobov 4f27c5f82b
Merge pull request #4768 from maranqz/feature/4606-exporter-from-FEEDS 
Pass info from FEEDS to ItemExporter
 2020-10-02 00:10:05 +05:00
Adrián Chaves 8f46e84519 Fix style issues 2020-09-21 23:28:16 +02:00
Adrián Chaves 35726da434 tests/test_feedexport.py: remove unused import 2020-09-21 22:55:25 +02:00
Adrián Chaves 17e135377a Use mocking for tests/test_feedexport.py::BatchDeliveriesTest::test_s3_export 2020-09-21 22:54:39 +02:00
Adrián Chaves 56f05fb164 Use mocking for tests/test_feedexport.py::S3FeedStorageTest::test_store 2020-09-21 22:01:09 +02:00
Andrey Rahmatullin 5e997587d9
Remove dead boto2 code, deprecate is_botocore() (#4776) 2020-09-20 15:06:46 +02:00
maranqz 71d2c2f1a3 improve view of dict 2020-08-30 12:43:44 +03:00
maranqz a8e895e684 kwargs for Item exporters classes 
test
docs
 2020-08-30 10:57:22 +03:00
Ammar Najjar 7597193dbe
Merge branch 'master' into 4307-use-f-strings 
Resolve Conflicts:
	tests/test_middleware.py
 2020-08-26 13:46:46 +02:00
nyov cf50561b86
Allow passing classes directly in Settings (#3873) 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-08-26 13:08:14 +02:00
Ammar Najjar 58ca8bbf6d Use f-strings (#4307) 2020-08-26 07:11:05 +02:00
Adrián Chaves e70975f0bb
Allow overwriting feeds (#4512) 
Co-authored-by: Yuval Hager <yhager@yhager.com>
 2020-08-17 15:10:08 +02:00
Mikhail Korobov 61459de476
Merge pull request #4688 from ashellunts/obsolete-s3feednokeys 
Remove obsolete S3FeedStorage instancing without AWS credentials
 2020-08-17 13:58:32 +05:00
Adrián Chaves 5138f9a965 Merge remote-tracking branch 'upstream/master' into allow-customizing-export-column-names 2020-08-11 13:42:05 +02:00
Artur Shellunts 5ec66be4a4
Merge branch 'master' into obsolete-s3feednokeys 2020-07-30 22:18:00 +02:00
Adrián Chaves 890b2138a6 Remove the u prefix from strings 2020-07-30 13:39:30 +02:00
Artur Shellunts 430d22e46e Remove not used import warnings 2020-07-21 23:39:04 +02:00
nyov 2829cd4268 Allow use without credentials 2020-07-21 23:38:02 +02:00
nyov 98e8086d1b Adapt S3FeedStorage testcase 2020-07-21 23:38:02 +02:00
nyov ece4fa6c7c Fix ignored testcase: boto is never installed 2020-07-21 23:38:01 +02:00
BroodingKangaroo 3e0492741d Another try to fix test errors on Windows 2020-07-19 00:10:29 +03:00
BroodingKangaroo 86f7ac2f2b Try to fix error at Windows 2020-07-17 18:22:47 +03:00
Adrián Chaves c4f9250220
Merge branch 'master' into ISSUE-4250-add_batch_deliveries 2020-07-16 17:44:18 +02:00
Mikhail Korobov 07470e1a3c
Merge pull request #3608 from ejulio/feat-685 
[MRG+1] Fix for #685 Add Google Cloud Storage Feed Export
 2020-07-16 18:11:19 +05:00
BroodingKangaroo 6454d456d2 Make check of placeholder less strict 2020-07-03 10:24:13 +03:00
Adrián Chaves 3199048520 Complete Azure Pipelines CI setup 2020-07-03 00:56:24 +02:00
BroodingKangaroo 1e245046ed Change setting name. Add leading zeroes to batch_id. Minor fixes. 2020-07-02 12:38:08 +03:00