RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
8,289 Commits 42 Branches 132 Tags 168 MiB
Commit Graph

4921 Commits

Author SHA1 Message Date
Jose Galdos 13181ba788 Improve http status all on http error middleware 2020-08-06 20:43:16 -05:00
Andrey Rahmatullin f3372a3753
Merge pull request #4254 from elacuesta/spider.parse 
Change Scraper API to call internal `_parse` method
 2020-07-21 17:37:54 +05:00
Akshay Sharma de297a3a16
enable ANSI color (instead of ANSI color codes) in the Windows terminal #4393 (#4403) 
* changed ie. -> i.e.(spelling error) on lines 667, 763 (issue scrapy#4332)

* updated all text files for issue #4332 (ie. -> i.e.)

* Apply ie. → i.e. in source comments

* ie → e.g.

* modified scrapy/utils/display.py to stop ANSI color sequences in the Windows terminal (issue #4393)

* modified scrapy/utils/display.py to stop ANSI color sequences in the Windows terminal (issue #4393)

* enabled virtual terminal processing (pr #4403)

* check for specific windows 10 version (pr #4403)

* fixing flake-8 test (pr #4403)

* added error handling for terminal info (pr #4403)

* corrected stderr (pr #4403)

* changed orientation, removed unwanted spaces (pr #4403)

* no need for style variable (pr #4403)

* fixing trailing whitespaces

* commenting windows check

* Update scrapy/utils/display.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update scrapy/utils/display.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update scrapy/utils/display.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update scrapy/utils/display.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* small fixes

* Shifting _color_support_info() function

* enabled virtual terminal processing (pr #4403)

* check for specific windows 10 version (pr #4403)

* fixing flake-8 test (pr #4403)

* added error handling for terminal info (pr #4403)

* corrected stderr (pr #4403)

* changed orientation, removed unwanted spaces (pr #4403)

* no need for style variable (pr #4403)

* fixing trailing whitespaces

* commenting windows check

* Update scrapy/utils/display.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update scrapy/utils/display.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update scrapy/utils/display.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update scrapy/utils/display.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* small fixes

* Shifting _color_support_info() function

* error handling

* error handlingy

* raise ValueError

* added in-built function for version comparison

* recommit changes

* changed check -> parse

* version comparison -> parse_version

* added scrapy/utils/display.py in pytest.ini

* Trigger

* Add simple test for scrapy.utils.display._colorize

* Flake8: E501 for tests/test_utils_display.py

* assertEquals -> assertEqual

* Normal formatter for all platforms

* separate test for windows

* all curses under try block

* added global TestStr

* more test added

* small fix

* covering exceptions

* windows test failing

* Refactor output color handling

* Fix pprint test

* fix flake8

Co-authored-by: Adrián Chaves <adrian@chaves.io>
Co-authored-by: Eugenio Lacuesta <eugenio.lacuesta@gmail.com>
 2020-07-20 17:23:38 +05:00
Artur Shellunts 62a4ede5e9
Remove deprecated classes BaseSgmlLinkExtractor, RegexLinkExtractor and SgmlLinkExtractor (#4356) 2020-07-17 12:40:04 +02:00
Mikhail Korobov 9a74a71c63
Merge pull request #4682 from noviluni/remove_python2_reminiscence 
remove python 2 reminiscence in cookies
 2020-07-16 23:58:58 +05:00
Marc Hernández b97a39fda0
deprecate retry_on_eintr (#4683) 2020-07-16 17:38:22 +02:00
Mikhail Korobov 07470e1a3c
Merge pull request #3608 from ejulio/feat-685 
[MRG+1] Fix for #685 Add Google Cloud Storage Feed Export
 2020-07-16 18:11:19 +05:00
Mikhail Korobov 0f2f1acf04
Merge pull request #4458 from scrapy/azure-pipelines 
Set up CI with Azure Pipelines
 2020-07-16 18:09:31 +05:00
Marc 0e0d1ad643 remove python 2 reminiscence in cookies 2020-07-16 14:25:46 +02:00
Júlio César Batista 38496a00b7
Use the itemlaoders library (#4516) 2020-07-15 12:08:36 +02:00
Andrey Rahmatullin eb1bc74417
Merge pull request #4674 from ashellunts/delete_deprecated_htmlparserlinkextractor 
Remove deprecated class HtmlParserLinkExtractor
 2020-07-15 12:51:42 +05:00
Adrián Chaves 6e119bd3e3
Merge branch 'master' into azure-pipelines 2020-07-14 11:26:19 +02:00
Adrián Chaves 53c323b19d _is_path → _is_filesystem_path 2020-07-13 15:29:30 +02:00
Adrián Chaves d54c4496ee Refactor guess_scheme 2020-07-13 14:37:08 +02:00
Andrey Rahmatullin 0ebba175ea
Merge pull request #4666 from Gallaecio/execution-access 
Fix permission handling on project generation from template files
 2020-07-13 16:01:06 +05:00
Andrey Rahmatullin b6f8693db9
Merge pull request #4673 from elacuesta/remove-backslash 
Remove backslash
 2020-07-13 11:14:00 +05:00
Aditya Kumar 3f7e8635f4
Allow the parse command to write data to a file (#4377) 2020-07-11 08:48:24 +02:00
Artur Shellunts a6a5fa91da Remove deprecated class HtmlParserLinkExtractor 
Issue #4356
 2020-07-10 23:10:49 +02:00
Eugenio Lacuesta 9e99be982a
Remove backslash 2020-07-09 09:51:28 -03:00
Eugenio Lacuesta 2be2bdd2df
Merge remote-tracking branch 'upstream/master' into spider.parse 2020-07-08 14:00:54 -03:00
Adrián Chaves ca77ca1f75 Generate read-only files on the fly 2020-07-07 14:44:03 +02:00
Adrián Chaves 79b4dfc53e Fix permission handling on project generation from template files 2020-07-07 14:07:04 +02:00
Adrián Chaves 6e58da1dcd
Merge branch 'master' into azure-pipelines 2020-07-02 17:49:42 +02:00
Marc Hernández 464f24f8c1
Add --data-raw to utils.curl and fix missing method with data (#4612) 2020-06-29 14:20:29 +02:00
Adrián Chaves 9f60481360 Bump version: 2.1.0 → 2.2.0 2020-06-24 12:27:39 +02:00
Adrián Chaves 6f4c964aa4
Cover Scrapy 2.2.0 in the release notes (#4630) 2020-06-24 12:26:38 +02:00
Eugenio Lacuesta c7f1c7e3f3
Merge branch 'master' into feat-685 2020-06-19 21:30:54 -03:00
Stanislau Hluboki a4bfd5ab6f Fix duplicated feed logs 2020-06-19 17:51:56 +03:00
Stas Glubokiy 3d027fb578
Fix missing storage.store calls in FeedExporter.close_spider (#4626) 2020-06-17 17:08:14 +02:00
Ram Rachum 214da8e523
Use chain.from_iterable in python.py 2020-06-17 13:50:54 +03:00
Eugenio Lacuesta ec98dabfab
Support for dataclass and attrs items (#3881) 2020-06-14 11:45:27 +02:00
Eugenio Lacuesta 8b549392f9
Bump minimum Python version to 3.5.2 (#4615) 2020-06-11 14:53:59 +02:00
Adrián Chaves 76a2cbf0ff
Apply minor style changes 2020-06-09 21:30:19 +02:00
Matthias Meschede 2df3b54c7d refactor 2020-06-05 09:29:05 +02:00
Matthias Meschede 0cabf406d5 set write permission to startproject folder 2020-06-04 10:39:40 +02:00
Andrey Rahmatullin 63929e75e3
Merge pull request #4552 from jay24rajput/common_commands 
Code sharing between crawl and runspider command
 2020-06-03 12:06:12 +05:00
Eugenio Lacuesta 91e505edfe
Return single element from coroutine callback (#4609) 2020-06-02 10:32:14 +02:00
Bulat Khabibullin 5cef927944
Implement TextResponse.json() (#4574) 2020-06-01 06:57:23 +02:00
Mikhail Korobov 6aab3badfa
Merge pull request #4568 from elacuesta/remove-monkeypatches-flake8-e402 
Remove _monkeypatches, import-related Flake8 codes
 2020-05-30 00:07:50 +05:00
Mikhail Korobov 13215cfb71
Merge pull request #4559 from elacuesta/cancel-request-download 
Stop response download from signal handler
 2020-05-27 21:46:00 +05:00
Andrey Rahmatullin ff699eb1f8
Merge pull request #4599 from elacuesta/LocalWeakReferencedCache-getitem-fix 
Fix KeyError in LocalWeakReferencedCache
 2020-05-27 21:42:04 +05:00
Eugenio Lacuesta 44d5801b05
Fix KeyError in LocalWeakReferencedCache 2020-05-27 11:15:33 -03:00
Eugenio Lacuesta 492197e477
StopDownload: store response in the exception instead of the failure 2020-05-27 10:45:07 -03:00
Eugenio Lacuesta f6ed5edc31
CookiesMiddleware: keep cookies from 'Cookie' request header 2020-05-21 11:07:41 -03:00
Eugenio Lacuesta 9514393bd1
Reduce amount of lines 2020-05-20 15:28:54 -03:00
Eugenio Lacuesta 276721a5dc
Stop response download 2020-05-20 14:46:26 -03:00
Mikhail Korobov febe82a907
Merge pull request #4486 from ilias-ant/add-file-status-on-media-pipelines-file-info 
Add status (downloaded, uptodate) to files information
 2020-05-18 22:30:30 +05:00
Mikhail Korobov bcc40c4077
better deprecation warning for Response.body_as_unicode() (#4579) 2020-05-18 19:12:31 +02:00
Eugenio Lacuesta a22f97052e
Cleanup import in selector module 2020-05-15 21:18:26 -03:00
Eugenio Lacuesta a915af2e45
Remove monkeypatches module, E402 flake8 code 2020-05-15 20:41:51 -03:00