RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
8,424 Commits 42 Branches 132 Tags 168 MiB
Commit Graph

4999 Commits

Author SHA1 Message Date
drs-11 ecec5f9e51 Cleaned up code 2020-08-25 02:46:44 +05:30
drs-11 1f9598ada4 Update branch to latest 2020-08-25 02:14:22 +05:30
drs-11 8b84a65a6b cleaned up code relating to issue #3689 2020-08-25 00:30:17 +05:30
Grisha Temchenko d68aab992e
Smarter generator check for combined return/yield statements (#4721) 2020-08-20 10:22:07 -03:00
sakshamb2113 42383cc267
Add a setting to customize the asyncio event loop (#4414) 2020-08-19 09:18:14 +02:00
Adrián Chaves e70975f0bb
Allow overwriting feeds (#4512) 
Co-authored-by: Yuval Hager <yhager@yhager.com>
 2020-08-17 15:10:08 +02:00
Mikhail Korobov 61459de476
Merge pull request #4688 from ashellunts/obsolete-s3feednokeys 
Remove obsolete S3FeedStorage instancing without AWS credentials
 2020-08-17 13:58:32 +05:00
Adrián Chaves 792d29380e
Merge Twisted’s deprecated HTTPClientFactory into ScrapyHTTPClientFactory (#4165) 
* Port code from Twisted’s deprecated HTTPClientFactory into ScrapyHTTPClientFactory

* Mention that ScrapyHTTPClientFactory has Twisted code
 2020-08-17 10:54:58 +02:00
Grammy Jiang 55edf8d3b8
Add typing hint to httpcache downloadermiddlewares (#4243) 2020-08-17 10:50:52 +02:00
Ajay Mittur a8e08d51cd
Check if file is already present on running `scrapy genspider` and terminate if so (#4623) 2020-08-17 10:45:52 +02:00
Eugenio Lacuesta 2aa4f3cbf9
Conditional request attribute binding for responses (#4632) 2020-08-17 10:39:59 +02:00
Mikhail Korobov 61653418ff
Merge pull request #4090 from Gallaecio/documentation-build 
Fix references to Python types in parameter type fields
 2020-08-14 00:58:06 +05:00
Ajay Mittur 1c4b4cc6b0
Support defining file path based on item in media pipelines (#4686) 2020-08-11 14:12:44 +02:00
Adrián Chaves b2f4df5cb7 Merge remote-tracking branch 'upstream/master' into documentation-build 2020-08-11 13:16:14 +02:00
Andrey Rahmatullin 0cf1340c29
Merge pull request #4663 from ajaymittur28/scrapy-check-test 
Add Tests to `scrapy check` command
 2020-08-10 16:18:29 +05:00
Andrey Rahmatullin 702de0480b
Merge pull request #4564 from jacty/patch-1 
Unnecessary update when value is None
 2020-08-10 16:14:42 +05:00
Andrey Rahmatullin 1f0722c87e
Merge pull request #4701 from kshitijcode/weakkeycache-cleanup-4684 
Code cleanup scrapy.utils.python.WeakKeyCache #4684
 2020-08-05 17:48:41 +05:00
Kshitij Sharma 983b7ddf2e aesthetic fixes 2020-08-05 16:13:52 +05:30
Kshitij Sharma b35d1f2b2c deleted tester.py 2020-08-05 09:14:04 +05:30
Kshitij Sharma 9d84289109 deprecated weakkeycache by specifying in __init__ 2020-08-05 09:11:59 +05:30
Marc Hernández 336f19f5cc
Change super syntax (#4707) 2020-08-04 20:42:01 +02:00
Adrián Chaves 1278e76d90 Bump version: 2.2.0 → 2.3.0 2020-08-04 20:07:02 +02:00
Adrián Chaves 3600582f56
Cover Scrapy 2.2.1 and 2.3 in the release notes (#4708) 2020-08-04 20:05:56 +02:00
Artur Shellunts 5ec66be4a4
Merge branch 'master' into obsolete-s3feednokeys 2020-07-30 22:18:00 +02:00
Adrián Chaves 890b2138a6 Remove the u prefix from strings 2020-07-30 13:39:30 +02:00
Kshitij Sharma 49337bd2ae Code cleanup scrapy.utils.python.WeakKeyCache #4684 and fixing ci alerts 2020-07-30 12:25:21 +05:30
Mikhail Korobov 5e2d1bd187
Merge pull request #4434 from BroodingKangaroo/ISSUE-4250-add_batch_deliveries 
Feed exports: add batch deliveries
 2020-07-29 23:03:28 +05:00
Kshitij Sharma e7a58fe157 Code cleanup scrapy.utils.python.WeakKeyCache #4684 2020-07-29 10:16:18 +05:30
Eugenio Lacuesta 5265853937
Use ItemAdapter.field_names when writing header in CsvItemExporter (#4668) 2020-07-28 11:15:14 +02:00
nyov 2829cd4268 Allow use without credentials 2020-07-21 23:38:02 +02:00
nyov 234c8b8c50 Removing deprecated S3FeedStorage without AWS keys instancing. 2020-07-21 23:38:01 +02:00
Andrey Rahmatullin f3372a3753
Merge pull request #4254 from elacuesta/spider.parse 
Change Scraper API to call internal `_parse` method
 2020-07-21 17:37:54 +05:00
Akshay Sharma de297a3a16
enable ANSI color (instead of ANSI color codes) in the Windows terminal #4393 (#4403) 
* changed ie. -> i.e.(spelling error) on lines 667, 763 (issue scrapy#4332)

* updated all text files for issue #4332 (ie. -> i.e.)

* Apply ie. → i.e. in source comments

* ie → e.g.

* modified scrapy/utils/display.py to stop ANSI color sequences in the Windows terminal (issue #4393)

* modified scrapy/utils/display.py to stop ANSI color sequences in the Windows terminal (issue #4393)

* enabled virtual terminal processing (pr #4403)

* check for specific windows 10 version (pr #4403)

* fixing flake-8 test (pr #4403)

* added error handling for terminal info (pr #4403)

* corrected stderr (pr #4403)

* changed orientation, removed unwanted spaces (pr #4403)

* no need for style variable (pr #4403)

* fixing trailing whitespaces

* commenting windows check

* Update scrapy/utils/display.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update scrapy/utils/display.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update scrapy/utils/display.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update scrapy/utils/display.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* small fixes

* Shifting _color_support_info() function

* enabled virtual terminal processing (pr #4403)

* check for specific windows 10 version (pr #4403)

* fixing flake-8 test (pr #4403)

* added error handling for terminal info (pr #4403)

* corrected stderr (pr #4403)

* changed orientation, removed unwanted spaces (pr #4403)

* no need for style variable (pr #4403)

* fixing trailing whitespaces

* commenting windows check

* Update scrapy/utils/display.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update scrapy/utils/display.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update scrapy/utils/display.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update scrapy/utils/display.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* small fixes

* Shifting _color_support_info() function

* error handling

* error handlingy

* raise ValueError

* added in-built function for version comparison

* recommit changes

* changed check -> parse

* version comparison -> parse_version

* added scrapy/utils/display.py in pytest.ini

* Trigger

* Add simple test for scrapy.utils.display._colorize

* Flake8: E501 for tests/test_utils_display.py

* assertEquals -> assertEqual

* Normal formatter for all platforms

* separate test for windows

* all curses under try block

* added global TestStr

* more test added

* small fix

* covering exceptions

* windows test failing

* Refactor output color handling

* Fix pprint test

* fix flake8

Co-authored-by: Adrián Chaves <adrian@chaves.io>
Co-authored-by: Eugenio Lacuesta <eugenio.lacuesta@gmail.com>
 2020-07-20 17:23:38 +05:00
Artur Shellunts 62a4ede5e9
Remove deprecated classes BaseSgmlLinkExtractor, RegexLinkExtractor and SgmlLinkExtractor (#4356) 2020-07-17 12:40:04 +02:00
Mikhail Korobov 9a74a71c63
Merge pull request #4682 from noviluni/remove_python2_reminiscence 
remove python 2 reminiscence in cookies
 2020-07-16 23:58:58 +05:00
Adrián Chaves c4f9250220
Merge branch 'master' into ISSUE-4250-add_batch_deliveries 2020-07-16 17:44:18 +02:00
Marc Hernández b97a39fda0
deprecate retry_on_eintr (#4683) 2020-07-16 17:38:22 +02:00
Mikhail Korobov 07470e1a3c
Merge pull request #3608 from ejulio/feat-685 
[MRG+1] Fix for #685 Add Google Cloud Storage Feed Export
 2020-07-16 18:11:19 +05:00
Mikhail Korobov 0f2f1acf04
Merge pull request #4458 from scrapy/azure-pipelines 
Set up CI with Azure Pipelines
 2020-07-16 18:09:31 +05:00
Marc 0e0d1ad643 remove python 2 reminiscence in cookies 2020-07-16 14:25:46 +02:00
Júlio César Batista 38496a00b7
Use the itemlaoders library (#4516) 2020-07-15 12:08:36 +02:00
Andrey Rahmatullin eb1bc74417
Merge pull request #4674 from ashellunts/delete_deprecated_htmlparserlinkextractor 
Remove deprecated class HtmlParserLinkExtractor
 2020-07-15 12:51:42 +05:00
Adrián Chaves 6e119bd3e3
Merge branch 'master' into azure-pipelines 2020-07-14 11:26:19 +02:00
Adrián Chaves 53c323b19d _is_path → _is_filesystem_path 2020-07-13 15:29:30 +02:00
Adrián Chaves d54c4496ee Refactor guess_scheme 2020-07-13 14:37:08 +02:00
Andrey Rahmatullin 0ebba175ea
Merge pull request #4666 from Gallaecio/execution-access 
Fix permission handling on project generation from template files
 2020-07-13 16:01:06 +05:00
Andrey Rahmatullin b6f8693db9
Merge pull request #4673 from elacuesta/remove-backslash 
Remove backslash
 2020-07-13 11:14:00 +05:00
Aditya Kumar 3f7e8635f4
Allow the parse command to write data to a file (#4377) 2020-07-11 08:48:24 +02:00
Artur Shellunts a6a5fa91da Remove deprecated class HtmlParserLinkExtractor 
Issue #4356
 2020-07-10 23:10:49 +02:00
Eugenio Lacuesta 9e99be982a
Remove backslash 2020-07-09 09:51:28 -03:00