RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
9,472 Commits 42 Branches 132 Tags 168 MiB
Commit Graph

90 Commits

Author SHA1 Message Date
Mikhail Korobov 500dae82e2
Merge pull request #3696 from Gallaecio/allow-customizing-export-column-names 
Allow customizing export column names
 2022-06-27 00:26:05 +05:00
Michel Ace 6e878490e8
Support and prefer the .jsonl file extension (#4848) 2022-06-17 08:37:14 +02:00
Adrián Chaves e8503217fb Merge remote-tracking branch 'upstream/master' into allow-customizing-export-column-names 2022-06-16 20:19:52 +02:00
Adrián Chaves aa0306a167
Cover 2.6.0 in the release notes (#5399) 2022-03-01 12:16:37 +01:00
Sixuan (Cherie) Wu 30d5779ea9
Fix FEED_URI_PARAMS: custom params throws KeyError (#4966) 
* fix FEED_URI_PARAMS: custom params throws KeyError closes #4962

* another try FEED_URI_PARAMS

* add warning message and change default function

* Add tests for FEED_URI_PARAMS

* FEED_URI_PARAMS: warn if the params dict has been modified in-place

* [Doc] FEED_URI_PARAMS: modifying params in-place is deprecated

* Remove whileline

* Rename parameters for lambda function

* Type hints for FeedExporter._get_uri_params

Co-authored-by: Adrián Chaves <adrian@chaves.io>
Co-authored-by: Eugenio Lacuesta <eugenio.lacuesta@gmail.com>
 2022-01-28 14:30:30 -03:00
D R Siddhartha 43ea21e830
Feed post-processing plugin support (#5190) 2021-08-24 11:48:01 +02:00
laggardkernel 8e7b96d8a2 Tweak doc for setting AWS_SESSION_TOKEN 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-07-27 19:40:02 +08:00
laggardkernel 7e4321f201 Add support for temporary security credential in AWS auth 2021-07-27 19:39:59 +08:00
Marlena Chatzigrigoriou d7deba7e89
Document all import paths and use the shortest in examples (#5099) 2021-07-14 10:34:28 +02:00
Türkalp Burak KAYRANCIOĞLU fcc6becc58
S3FeedStorage: allow custom endpoint (#4998) 
Co-authored-by: Andrey Rahmatullin <wrar@wrar.name>
 2021-07-14 10:00:43 +02:00
D R Siddhartha 4ddc9d6b55
Feeds: Item Filters (#5178) 2021-07-13 17:22:29 +02:00
Shinichi Takayanagi bd60c3f41f
More documentation for setting spider atributes 
* docs: require sphinx-rtd-theme>=0.5.2 and the latest pip to prevent installing breaking docutils>=0.17

* Update feed-exports.rst

* Update feed-exports.rst

* Reflects the comments

* Remove redundant newline

* Update docs/topics/feed-exports.rst

Co-authored-by: Adrián Chaves <adrian@chaves.io>

* Apply suggestions from code review

Co-authored-by: Adrián Chaves <adrian@chaves.io>

Co-authored-by: Adrián Chaves <adrian@chaves.io>
Co-authored-by: Eugenio Lacuesta <eugenio.lacuesta@gmail.com>
 2021-05-11 00:58:04 +05:00
Adrián Chaves 1773eaf5dc Move lists closer to their introducing paragraph 2021-02-03 11:43:18 +01:00
Adrián Chaves 07379cf9b7
Merge branch 'master' into allow-customizing-export-column-names 2020-11-20 20:06:18 +01:00
Adrián Chaves 015c82b974
Scrapy 2.4 release notes (#4808) 2020-10-11 22:09:45 +02:00
Ilia Sergunin d10464ca96
Update docs/topics/feed-exports.rst 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-09-01 10:13:40 +03:00
maranqz a8e895e684 kwargs for Item exporters classes 
test
docs
 2020-08-30 10:57:22 +03:00
Adrián Chaves 3f0a677c04
Cover version directive usage in the documentation policy (#4310) 
* Cover version directives in the documentation policy

* Remove version directives in preparation for Scrapy 2.0

* Update the policy based on the deprecation policy

* Only remove version directives after 3 years
 2020-08-27 23:56:58 +05:00
Adrián Chaves e70975f0bb
Allow overwriting feeds (#4512) 
Co-authored-by: Yuval Hager <yhager@yhager.com>
 2020-08-17 15:10:08 +02:00
Adrián Chaves 65e0abaea5 Document FEED_URI_PARAMS 2020-08-13 09:05:51 +02:00
Adrián Chaves 5138f9a965 Merge remote-tracking branch 'upstream/master' into allow-customizing-export-column-names 2020-08-11 13:42:05 +02:00
Adrián Chaves 3600582f56
Cover Scrapy 2.2.1 and 2.3 in the release notes (#4708) 2020-08-04 20:05:56 +02:00
BroodingKangaroo a6c1d79b7c pep8 tiny changes 2020-07-28 11:53:05 +03:00
Adrián Chaves c4f9250220
Merge branch 'master' into ISSUE-4250-add_batch_deliveries 2020-07-16 17:44:18 +02:00
BroodingKangaroo 41263f61c6 Change single quotes to double in example in docs 2020-07-16 18:41:45 +03:00
BroodingKangaroo 8bdcdb0a76 Add quotes to example in docs 2020-07-16 09:13:54 +03:00
BroodingKangaroo 6454d456d2 Make check of placeholder less strict 2020-07-03 10:24:13 +03:00
BroodingKangaroo 1e245046ed Change setting name. Add leading zeroes to batch_id. Minor fixes. 2020-07-02 12:38:08 +03:00
BroodingKangaroo 7b1d3c35ea Minor updates 2020-07-01 12:24:31 +03:00
BroodingKangaroo 05c2587c6a Docs update and tiny fixes 2020-06-28 09:45:45 +03:00
BroodingKangaroo c3cee74fd4 Change default value of FEED_STORAGE_BATCH_ITEM_COUNT to 0 2020-06-26 18:45:21 +03:00
BroodingKangaroo 77333666f1 resolve conflicts 2020-06-26 14:55:46 +03:00
Eugenio Lacuesta c7f1c7e3f3
Merge branch 'master' into feat-685 2020-06-19 21:30:54 -03:00
Eugenio Lacuesta ec98dabfab
Support for dataclass and attrs items (#3881) 2020-06-14 11:45:27 +02:00
BroodingKangaroo 10ae1a284f Minor fixes 2020-05-15 22:50:54 +03:00
BroodingKangaroo 1cdcf8b08b Minor fixes 2020-05-15 20:14:07 +03:00
BroodingKangaroo 69c005f013 Documentation indent fix 2020-05-14 10:35:56 +03:00
BroodingKangaroo 111a58fe3d Merge remote-tracking branch 'upstream/master' into ISSUE-4250-add_batch_deliveries 2020-05-14 09:53:14 +03:00
BroodingKangaroo 8662d3587d Documentation and code refactoring 2020-05-14 00:05:36 +03:00
BroodingKangaroo 2327ecead0 Rename FEED_STORAGE_BATCH_SIZE to FEED_STORAGE_BATCH_ITEM_COUNT 2020-05-13 22:50:04 +03:00
Adrián Chaves a859ea0c8b Merge remote-tracking branch 'upstream/master' into allow-customizing-export-column-names 2020-05-06 19:48:32 +02:00
BroodingKangaroo dad2ea7522 Change time_id to batch_time 2020-05-02 01:21:03 +03:00
BroodingKangaroo 2eee6c8101 Documentation spelling fix 2020-04-27 09:58:14 +03:00
BroodingKangaroo f0f1be76d1 Using time_id instead of time as a timestamp 2020-04-27 09:56:57 +03:00
BroodingKangaroo cac1f3a6ad Update documentation 2020-04-16 10:06:56 +03:00
Eugenio Lacuesta 34d607194b
Merge branch 'master' into feat-685 2020-03-18 21:24:36 -03:00
Eugenio Lacuesta 49156f2ecb
[doc] Feed exports: full local path as example 2020-03-11 20:45:54 -03:00
Eugenio Lacuesta ada37c5409
Export to multiple formats in a single crawl 2020-03-05 09:33:57 -03:00
Adrián Chaves 6aa0ba4553
Write release notes for Scrapy 2.0.0 (#4329) 2020-03-03 09:11:11 +01:00
Akshay Sharma 182445f9d9
Fix a spelling error: ie. → i.e. (#4338) 2020-02-18 17:58:31 +01:00