RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,119 Commits 49 Branches 132 Tags 132 MiB
Commit Graph

183 Commits

Author SHA1 Message Date
Laerte Pereira b06936f111 Handle Tuple type on getdictorlist method, bump 3.12 python version 2023-08-14 10:33:48 -03:00
Andrey Rakhmatullin 644a71bfd4 Use ftp:// URLs in FTP tests. 2023-08-07 00:17:52 +04:00
Laerte Pereira b9c32a0cfd
Remove functions deprecated in 2.6.0 (#5996) 2023-08-03 17:06:55 +02:00
Kevin Lloyd Bernal 9a1bf40c2f expose AWS_REGION_NAME in S3FeedStorage 2023-07-24 16:37:52 +08:00
Mikhail Korobov af2aa4b421
Merge pull request #5847 from namelessGonbai/feat/FixFeedExport 
Fix FeedExporter not to export empty file
 2023-07-22 23:18:36 +05:00
Andrey Rakhmatullin e058a05763 Skip tests that don't work on Python 3.12. 2023-07-22 17:51:13 +04:00
Alex 04ee3303e4
Adding support for Windows of absolute pathlib.Path objects in FeedExporter (#5939) 2023-06-22 07:04:06 +02:00
Andrey Rakhmatullin 85fe88f80f
Merge pull request #5833 from GodelTech/issue_960_s3_feed_export 
Fix #960: S3 Feed Export throws boto error
 2023-06-13 18:44:06 +04:00
Andrey Rakhmatullin 8a64f3e8de
Merge branch 'master' into feat/FixFeedExport 2023-06-13 18:21:06 +04:00
guillermo-bondonno b50c032ee9
Add feed_slot_closed and feed_exporter_closed signals (#5876) 2023-04-26 08:20:37 +02:00
tstauder ef61fb5698
Fix flaky tests involving feed export batches (#5898) 
Co-authored-by: jmannoop <jmannoop@umich.edu>
 2023-04-19 08:33:32 +02:00
Andrey Rakhmatullin 1e20ba0a1b
Merge branch 'master' into feat/FixFeedExport 2023-04-11 19:48:19 +04:00
Felipe Boff Nunes a0e2e36b52 adjustments 2023-04-06 14:23:19 -03:00
Felipe Boff Nunes 0ffd1667ba adjustments 2023-04-06 14:17:35 -03:00
jazzthief 4ebc08ef10
Switch from `is_boto3_available()` to `IS_BOTO3_AVAILABLE` var 2023-03-16 17:24:11 +01:00
kenshi kikuchi c8ed793257 Fix test_export_no_items_multiple_feeds 2023-03-16 17:16:14 +09:00
Adrián Chaves 7e7b41c6b3 Fix test expectations for boto3 2023-03-15 16:38:13 +01:00
Adrián Chaves 39dbfa1d82 Minimize test reliance on S3; do not install botocore on the default test environments 2023-03-15 15:31:39 +01:00
kenshi kikuchi d311779887 Fix FeedExporter 
+ Fix FeedExporter not to export empty file
+ Change default value of FEED_STORE_EMPTY
 2023-03-08 16:24:09 +09:00
Adrián Chaves b07d3f85a3 Upgrade CI tools 2023-02-02 06:37:40 +01:00
Jalil SA a5c1ef8276 sort imports with isort 2023-01-25 14:25:15 -06:00
Emmanuel Rondan 8ee4817471 reaplying black, fixing conflicts and ignoring bandit checks on test directory 2023-01-20 10:55:46 -03:00
Emmanuel Rondan e211ec0aa2 adding black formatter to all the code 2022-11-29 11:30:46 -03:00
Matthew Donoughe 7daf735f45
downgrade type hint syntax 2022-10-18 20:07:29 -04:00
Matthew Donoughe 40d9ca3bdd
use pathlib 2022-10-17 17:40:10 -04:00
Laerte Pereira 3ca7877781
chore: Skip `batch_path_differ` test on Windows 2022-09-29 11:51:50 -03:00
Adrián Chaves d599fff2b9 Test the life cycle of a storage file 2022-07-29 19:12:20 +02:00
Mikhail Korobov b103664bf4
Address 2/3 of warnings from tests (#5561) 2022-07-19 17:39:26 +02:00
Adrián Chaves ec5cf3e9ce test_feedexport: solve ordered comparison issues 2022-06-17 09:10:18 +02:00
Adrián Chaves bc285f393c Revert "Revert "test_feedexport: remove ordered=False"" 
This reverts commit 516e2d6ec0.
 2022-06-17 09:00:39 +02:00
Adrián Chaves 516e2d6ec0 Revert "test_feedexport: remove ordered=False" 
This reverts commit 24f382fa45.
 2022-06-17 08:55:45 +02:00
Adrián Chaves 3729c6d266 Remove unused import and redundant import 2022-06-17 08:33:34 +02:00
Adrián Chaves 24f382fa45 test_feedexport: remove ordered=False 2022-06-17 08:31:45 +02:00
Adrián Chaves 1b9ed22bec Remove Python < 3.7 leftover 2022-06-17 08:27:17 +02:00
Adrián Chaves 3e994bda45 Update for Python 3.7+ 2022-06-16 20:53:14 +02:00
Adrián Chaves e8503217fb Merge remote-tracking branch 'upstream/master' into allow-customizing-export-column-names 2022-06-16 20:19:52 +02:00
Sixuan (Cherie) Wu 30d5779ea9
Fix FEED_URI_PARAMS: custom params throws KeyError (#4966) 
* fix FEED_URI_PARAMS: custom params throws KeyError closes #4962

* another try FEED_URI_PARAMS

* add warning message and change default function

* Add tests for FEED_URI_PARAMS

* FEED_URI_PARAMS: warn if the params dict has been modified in-place

* [Doc] FEED_URI_PARAMS: modifying params in-place is deprecated

* Remove whileline

* Rename parameters for lambda function

* Type hints for FeedExporter._get_uri_params

Co-authored-by: Adrián Chaves <adrian@chaves.io>
Co-authored-by: Eugenio Lacuesta <eugenio.lacuesta@gmail.com>
 2022-01-28 14:30:30 -03:00
D R Siddhartha 43ea21e830
Feed post-processing plugin support (#5190) 2021-08-24 11:48:01 +02:00
Mannan2812 94baa4b272
Fix FileFeedStoragePreFeedOptionsTest fails in CI/CD pipeline (#5198) 2021-08-05 21:23:11 +02:00
laggardkernel 7e4321f201 Add support for temporary security credential in AWS auth 2021-07-27 19:39:59 +08:00
Türkalp Burak KAYRANCIOĞLU fcc6becc58
S3FeedStorage: allow custom endpoint (#4998) 
Co-authored-by: Andrey Rahmatullin <wrar@wrar.name>
 2021-07-14 10:00:43 +02:00
D R Siddhartha 4ddc9d6b55
Feeds: Item Filters (#5178) 2021-07-13 17:22:29 +02:00
Vostretsov Nikita 23cfdb058e
Reducing amount of warnings during test run (#5162) 
* put flake8 options into separate file to remove pytest warnings

* remove ResourceLeaked warning in pypy

* suppress warnings from twisted

* ignore deprecation warnings here

* ignore deprecation warning in tests of deprecated methods

* ignore deprecation warnings here

* update test classes

* don`t use deprecated method call

* ignore deprecation warnings here

* proper warning class

* more selective ignoring

* Revert "don`t use deprecated method call"

This reverts commit 59216ab560.
 2021-05-28 14:45:06 +05:00
Adrián Chaves 07379cf9b7
Merge branch 'master' into allow-customizing-export-column-names 2020-11-20 20:06:18 +01:00
joaquin garmendia 85604e1078
Add failed and success count stats to feedstorage backends (#4850) 2020-11-11 21:16:01 +01:00
Andrey Rahmatullin 1aeda66435
Merge pull request #4804 from Gallaecio/mock-s3 
Use mocking for S3 tests that currently need server credentials
 2020-10-06 18:23:01 +05:00
Mikhail Korobov 4f27c5f82b
Merge pull request #4768 from maranqz/feature/4606-exporter-from-FEEDS 
Pass info from FEEDS to ItemExporter
 2020-10-02 00:10:05 +05:00
Adrián Chaves 8f46e84519 Fix style issues 2020-09-21 23:28:16 +02:00
Adrián Chaves 35726da434 tests/test_feedexport.py: remove unused import 2020-09-21 22:55:25 +02:00
Adrián Chaves 17e135377a Use mocking for tests/test_feedexport.py::BatchDeliveriesTest::test_s3_export 2020-09-21 22:54:39 +02:00