13181ba788
Improve http status all on http error middleware
2020-08-06 20:43:16 -05:00
f3372a3753
Merge pull request #4254 from elacuesta/spider.parse
...
Change Scraper API to call internal `_parse` method
2020-07-21 17:37:54 +05:00
de297a3a16
enable ANSI color (instead of ANSI color codes) in the Windows terminal #4393 ( #4403 )
...
* changed ie. -> i.e.(spelling error) on lines 667, 763 (issue scrapy#4332)
* updated all text files for issue #4332 (ie. -> i.e.)
* Apply ie. → i.e. in source comments
* ie → e.g.
* modified scrapy/utils/display.py to stop ANSI color sequences in the Windows terminal (issue #4393 )
* modified scrapy/utils/display.py to stop ANSI color sequences in the Windows terminal (issue #4393 )
* enabled virtual terminal processing (pr #4403 )
* check for specific windows 10 version (pr #4403 )
* fixing flake-8 test (pr #4403 )
* added error handling for terminal info (pr #4403 )
* corrected stderr (pr #4403 )
* changed orientation, removed unwanted spaces (pr #4403 )
* no need for style variable (pr #4403 )
* fixing trailing whitespaces
* commenting windows check
* Update scrapy/utils/display.py
Co-Authored-By: Adrián Chaves <adrian@chaves.io>
* Update scrapy/utils/display.py
Co-Authored-By: Adrián Chaves <adrian@chaves.io>
* Update scrapy/utils/display.py
Co-Authored-By: Adrián Chaves <adrian@chaves.io>
* Update scrapy/utils/display.py
Co-Authored-By: Adrián Chaves <adrian@chaves.io>
* small fixes
* Shifting _color_support_info() function
* enabled virtual terminal processing (pr #4403 )
* check for specific windows 10 version (pr #4403 )
* fixing flake-8 test (pr #4403 )
* added error handling for terminal info (pr #4403 )
* corrected stderr (pr #4403 )
* changed orientation, removed unwanted spaces (pr #4403 )
* no need for style variable (pr #4403 )
* fixing trailing whitespaces
* commenting windows check
* Update scrapy/utils/display.py
Co-Authored-By: Adrián Chaves <adrian@chaves.io>
* Update scrapy/utils/display.py
Co-Authored-By: Adrián Chaves <adrian@chaves.io>
* Update scrapy/utils/display.py
Co-Authored-By: Adrián Chaves <adrian@chaves.io>
* Update scrapy/utils/display.py
Co-Authored-By: Adrián Chaves <adrian@chaves.io>
* small fixes
* Shifting _color_support_info() function
* error handling
* error handlingy
* raise ValueError
* added in-built function for version comparison
* recommit changes
* changed check -> parse
* version comparison -> parse_version
* added scrapy/utils/display.py in pytest.ini
* Trigger
* Add simple test for scrapy.utils.display._colorize
* Flake8: E501 for tests/test_utils_display.py
* assertEquals -> assertEqual
* Normal formatter for all platforms
* separate test for windows
* all curses under try block
* added global TestStr
* more test added
* small fix
* covering exceptions
* windows test failing
* Refactor output color handling
* Fix pprint test
* fix flake8
Co-authored-by: Adrián Chaves <adrian@chaves.io>
Co-authored-by: Eugenio Lacuesta <eugenio.lacuesta@gmail.com>
2020-07-20 17:23:38 +05:00
62a4ede5e9
Remove deprecated classes BaseSgmlLinkExtractor, RegexLinkExtractor and SgmlLinkExtractor ( #4356 )
2020-07-17 12:40:04 +02:00
d29bec60d7
Upgrade PyPy for CI, and test both 3.5 (oldest) and 3.6 (newest) ( #4504 )
...
* Upgrade PyPy for CI, and test both 3.5 (oldest) and 3.6 (newest)
* Log a detailed error message to discover why MockServer is not working
* Go for all lines!
* Disable tests based on mitmproxy while running on PyPy
* Fix test_get_func_args for PyPy 3.6+
* Make testPayloadDefaultCiphers work regardless of OpenSSL default ciphers
* Crossing fingers…
* Rename: testPayloadDefaultCiphers → testPayloadDisabledCipher
* Test the PyPy version currently documented as the minimum required version
* Fix the PYPY_VERSION tag
* Update the documentation about supported PyPy versions
* Also test the latest 3.5 Python version with PyPy
* Fix the PYPY_VERSION value for the latest 3.5 version
* Use pinned dependencies for asyncio and PyPy tests against oldest supported Python versions
* Fix PyPy installation for the pypy3-pinned Tox environment
* Try installing Cython
* Maybe PyPy requires lxml 3.6.0?
* install.rst: minor clarification
* lxml 4.0.0 is required on PyPy
* Require setuptools 18.5+
* Revert "Require setuptools 18.5+"
This reverts commit 017ec33ac2
2020-07-17 02:19:24 +05:00
9a74a71c63
Merge pull request #4682 from noviluni/remove_python2_reminiscence
...
remove python 2 reminiscence in cookies
2020-07-16 23:58:58 +05:00
07470e1a3c
Merge pull request #3608 from ejulio/feat-685
...
[MRG+1] Fix for #685 Add Google Cloud Storage Feed Export
2020-07-16 18:11:19 +05:00
0f2f1acf04
Merge pull request #4458 from scrapy/azure-pipelines
...
Set up CI with Azure Pipelines
2020-07-16 18:09:31 +05:00
0e0d1ad643
remove python 2 reminiscence in cookies
2020-07-16 14:25:46 +02:00
38496a00b7
Use the itemlaoders library ( #4516 )
2020-07-15 12:08:36 +02:00
eb1bc74417
Merge pull request #4674 from ashellunts/delete_deprecated_htmlparserlinkextractor
...
Remove deprecated class HtmlParserLinkExtractor
2020-07-15 12:51:42 +05:00
ed5247ca4c
Remove htmlparser.py from tests/ignore.txt
2020-07-14 18:06:11 +02:00
6e119bd3e3
Merge branch 'master' into azure-pipelines
2020-07-14 11:26:19 +02:00
53c323b19d
_is_path → _is_filesystem_path
2020-07-13 15:29:30 +02:00
d54c4496ee
Refactor guess_scheme
2020-07-13 14:37:08 +02:00
0ebba175ea
Merge pull request #4666 from Gallaecio/execution-access
...
Fix permission handling on project generation from template files
2020-07-13 16:01:06 +05:00
b6f8693db9
Merge pull request #4673 from elacuesta/remove-backslash
...
Remove backslash
2020-07-13 11:14:00 +05:00
3f7e8635f4
Allow the parse command to write data to a file ( #4377 )
2020-07-11 08:48:24 +02:00
9aea1f0961
Remove backslash (tests)
2020-07-09 11:04:46 -03:00
2be2bdd2df
Merge remote-tracking branch 'upstream/master' into spider.parse
2020-07-08 14:00:54 -03:00
7e38615703
Remove unused import
2020-07-07 15:30:19 +02:00
ca77ca1f75
Generate read-only files on the fly
2020-07-07 14:44:03 +02:00
e1450799ce
Remove debug test case variable
2020-07-07 14:11:37 +02:00
a3afff4a0e
Fix style issue
2020-07-07 14:11:02 +02:00
79b4dfc53e
Fix permission handling on project generation from template files
2020-07-07 14:07:04 +02:00
17aec5944c
Update tests/CrawlerRunner/ip_address.py
...
Co-authored-by: Eugenio Lacuesta <1731933+elacuesta@users.noreply.github.com>
2020-07-06 10:47:25 +02:00
ec06cf79a6
Update tests/CrawlerRunner/ip_address.py
...
Co-authored-by: Eugenio Lacuesta <1731933+elacuesta@users.noreply.github.com>
2020-07-06 10:47:11 +02:00
eb93774256
TrackrefTestCase.test_get_oldest: protect from lack of precision
2020-07-03 01:41:47 +02:00
3199048520
Complete Azure Pipelines CI setup
2020-07-03 00:56:24 +02:00
6e58da1dcd
Merge branch 'master' into azure-pipelines
2020-07-02 17:49:42 +02:00
e46b47c365
Renew the localhost certificate for tests ( #4650 )
...
Validity
Not Before: Jun 28 12:54:15 2020 GMT
Not After : Jun 28 12:54:15 2021 GMT
Subject: C = IE, O = Scrapy, CN = localhost
2020-06-29 15:05:13 +02:00
464f24f8c1
Add --data-raw to utils.curl and fix missing method with data ( #4612 )
2020-06-29 14:20:29 +02:00
3672f5f988
Spider constructor expects name as argument, not start_urls.
...
Fixes #4644
2020-06-23 14:56:12 +02:00
c7f1c7e3f3
Merge branch 'master' into feat-685
2020-06-19 21:30:54 -03:00
a4bfd5ab6f
Fix duplicated feed logs
2020-06-19 17:51:56 +03:00
3d027fb578
Fix missing storage.store calls in FeedExporter.close_spider ( #4626 )
2020-06-17 17:08:14 +02:00
ec98dabfab
Support for dataclass and attrs items ( #3881 )
2020-06-14 11:45:27 +02:00
91e505edfe
Return single element from coroutine callback ( #4609 )
2020-06-02 10:32:14 +02:00
5cef927944
Implement TextResponse.json() ( #4574 )
2020-06-01 06:57:23 +02:00
13215cfb71
Merge pull request #4559 from elacuesta/cancel-request-download
...
Stop response download from signal handler
2020-05-27 21:46:00 +05:00
ff699eb1f8
Merge pull request #4599 from elacuesta/LocalWeakReferencedCache-getitem-fix
...
Fix KeyError in LocalWeakReferencedCache
2020-05-27 21:42:04 +05:00
7bf375098e
Merge pull request #2400 from elacuesta/keep_cookie_header
...
[MRG+1] CookiesMiddleware: keep cookies from 'Cookie' request header, fix encoding
2020-05-27 21:41:43 +05:00
44d5801b05
Fix KeyError in LocalWeakReferencedCache
2020-05-27 11:15:33 -03:00
492197e477
StopDownload: store response in the exception instead of the failure
2020-05-27 10:45:07 -03:00
634ad5ebca
Merge remote-tracking branch 'upstream/master' into cancel-request-download
2020-05-27 10:27:45 -03:00
5b2af85289
Link to issue describing troublesome pytest versions
2020-05-22 15:09:58 -04:00
f6ed5edc31
CookiesMiddleware: keep cookies from 'Cookie' request header
2020-05-21 11:07:41 -03:00
fbe915d9cb
StopDownload: check partial response contents
2020-05-20 20:20:30 -03:00
276721a5dc
Stop response download
2020-05-20 14:46:26 -03:00
eb8493f09c
Incompatible with pytest 5.4 and 5.4.1
2020-05-20 09:21:34 -04:00
Jose Galdos
Andrey Rahmatullin
Akshay Sharma
Artur Shellunts
Adrián Chaves
Mikhail Korobov
Marc
Júlio César Batista
Aditya Kumar
Eugenio Lacuesta
Lukas Anzinger
Stanislau Hluboki
Eugenio Lacuesta
Bulat Khabibullin
Kyle Altendorf