RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,710 Commits 49 Branches 132 Tags 144 MiB
Commit Graph

10710 Commits

Author SHA1 Message Date
Artur Shellunts 5ec66be4a4
Merge branch 'master' into obsolete-s3feednokeys 2020-07-30 22:18:00 +02:00
Aditya d707f8b5d9 docs: mention H2DownloadHandler in settings.rst 2020-07-30 18:06:21 +05:30
Adrián Chaves 6f4ccec567 Cover our deprecation policy in the documentation 2020-07-30 14:03:14 +02:00
Adrián Chaves 890b2138a6 Remove the u prefix from strings 2020-07-30 13:39:30 +02:00
Aditya a3fecaf07f test: fix host-name H2DownloadHandler tests 2020-07-30 15:45:27 +05:30
Kshitij Sharma 49337bd2ae Code cleanup scrapy.utils.python.WeakKeyCache #4684 and fixing ci alerts 2020-07-30 12:25:21 +05:30
Kshitij Sharma 33ddc3d4f3 Trigger CI 2020-07-30 09:47:08 +05:30
Mikhail Korobov 5e2d1bd187
Merge pull request #4434 from BroodingKangaroo/ISSUE-4250-add_batch_deliveries 
Feed exports: add batch deliveries
 2020-07-29 23:03:28 +05:00
Kshitij Sharma 403bc7020a Code cleanup scrapy.utils.python.WeakKeyCache #4684 and fixing ci alerts 2020-07-29 18:05:33 +05:30
Aditya 19f2b4b53d refactor: AcceptableProtocolsContextFactory 
- rename H2WrappedContextFactory to AcceptableProtocolsContextFactory
- AcceptableProtocolsContextFactory accepts an argument
acceptable_protocols which can be used to override the context factory
priority list of protocols during ALPN or NPN
 2020-07-29 17:25:59 +05:30
Aditya e8342996f6 test: H2DownloadHandler 
Following tests are skipped as Content-Length header not matching the
data received is considered as a ProtocolError
- test_download_broken_content_cause_data_loss
- test_download_broken_chunked_content_cause_data_loss
- test_download_broken_content_allow_data_loss
- test_download_broken_chunked_content_allow_data_loss
- test_download_broken_content_allow_data_loss_via_setting
- test_download_broken_chunked_content_allow_data_loss_via_setting

BREAKING CHANGES
The following tests currently fail
- test_content_length_zero_bodyless_post_request_headers
- test_host_header_seted_in_request_headers
- test_download_with_maxsize_very_large_file
 2020-07-29 13:51:01 +05:30
Aditya 92bec38591 feat: MethodNotAllowed405, Content-Length header 
- add tests to check for Content-Length header
- raise MethodNotAllowed405 when remote send 'HTTP/2.0 405 Method Not
Allowed'
 2020-07-29 13:49:14 +05:30
Kshitij Sharma e7a58fe157 Code cleanup scrapy.utils.python.WeakKeyCache #4684 2020-07-29 10:16:18 +05:30
Eugenio Lacuesta 5265853937
Use ItemAdapter.field_names when writing header in CsvItemExporter (#4668) 2020-07-28 11:15:14 +02:00
BroodingKangaroo a6c1d79b7c pep8 tiny changes 2020-07-28 11:53:05 +03:00
Mikhail Korobov ce0c25fc85
Merge pull request #4690 from elacuesta/typing-setup-remove-monkeypatches 
Remove monkeypatches module from mypy section in setup.cfg
 2020-07-23 17:56:08 +05:00
Eugenio Lacuesta 8fae3d5bb7
Remove monkeypatches module from mypy section in setup.cfg 2020-07-22 16:08:35 -03:00
Aditya 031bfc9c3b feat(wip): ScrapyH2Agent, ScrapyProxyH2Agent 2020-07-22 15:01:59 +05:30
Aditya 62ce842afc fix: multiple h2 connections to same uri 
- When multiple requests are sent to H2ConnectionPool to the same uri
while the connection is in connecting state -- multiple connections were
establised.
- Fixed the bug using a deque of all the request deferred's which fire
with the H2ClientProtocol (connection) instance when connection is
established
 2020-07-22 14:52:49 +05:30
Aditya 8252a6f8d8 fix: H2Agent not able to connect via SSL 
- add H2WrappedContextFactory class which wraps the context factory
passed to H2Agent and updates the SSL context acceptable protocols list
to only h2
 2020-07-22 14:52:49 +05:30
Aditya 9fffb801ed feat: H2Agent, H2ConnectionPool base implementation 2020-07-22 14:52:49 +05:30
Aditya 3685e99cca test: http2 connection timeout 2020-07-22 14:47:20 +05:30
Aditya 316620b517 chore: pass spider as argument for request method 
- download_maxsize and download_warnsize can now be extracted from the
spider directly and passed to the stream
- remove `partial` flag from the response as per RFC 7540 - Section
8.1.2.6
 2020-07-22 13:57:49 +05:30
Artur Shellunts 430d22e46e Remove not used import warnings 2020-07-21 23:39:04 +02:00
nyov 2829cd4268 Allow use without credentials 2020-07-21 23:38:02 +02:00
nyov 98e8086d1b Adapt S3FeedStorage testcase 2020-07-21 23:38:02 +02:00
nyov 234c8b8c50 Removing deprecated S3FeedStorage without AWS keys instancing. 2020-07-21 23:38:01 +02:00
nyov ece4fa6c7c Fix ignored testcase: boto is never installed 2020-07-21 23:38:01 +02:00
Andrey Rahmatullin f3372a3753
Merge pull request #4254 from elacuesta/spider.parse 
Change Scraper API to call internal `_parse` method
 2020-07-21 17:37:54 +05:00
Akshay Sharma de297a3a16
enable ANSI color (instead of ANSI color codes) in the Windows terminal #4393 (#4403) 
* changed ie. -> i.e.(spelling error) on lines 667, 763 (issue scrapy#4332)

* updated all text files for issue #4332 (ie. -> i.e.)

* Apply ie. → i.e. in source comments

* ie → e.g.

* modified scrapy/utils/display.py to stop ANSI color sequences in the Windows terminal (issue #4393)

* modified scrapy/utils/display.py to stop ANSI color sequences in the Windows terminal (issue #4393)

* enabled virtual terminal processing (pr #4403)

* check for specific windows 10 version (pr #4403)

* fixing flake-8 test (pr #4403)

* added error handling for terminal info (pr #4403)

* corrected stderr (pr #4403)

* changed orientation, removed unwanted spaces (pr #4403)

* no need for style variable (pr #4403)

* fixing trailing whitespaces

* commenting windows check

* Update scrapy/utils/display.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update scrapy/utils/display.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update scrapy/utils/display.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update scrapy/utils/display.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* small fixes

* Shifting _color_support_info() function

* enabled virtual terminal processing (pr #4403)

* check for specific windows 10 version (pr #4403)

* fixing flake-8 test (pr #4403)

* added error handling for terminal info (pr #4403)

* corrected stderr (pr #4403)

* changed orientation, removed unwanted spaces (pr #4403)

* no need for style variable (pr #4403)

* fixing trailing whitespaces

* commenting windows check

* Update scrapy/utils/display.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update scrapy/utils/display.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update scrapy/utils/display.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update scrapy/utils/display.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* small fixes

* Shifting _color_support_info() function

* error handling

* error handlingy

* raise ValueError

* added in-built function for version comparison

* recommit changes

* changed check -> parse

* version comparison -> parse_version

* added scrapy/utils/display.py in pytest.ini

* Trigger

* Add simple test for scrapy.utils.display._colorize

* Flake8: E501 for tests/test_utils_display.py

* assertEquals -> assertEqual

* Normal formatter for all platforms

* separate test for windows

* all curses under try block

* added global TestStr

* more test added

* small fix

* covering exceptions

* windows test failing

* Refactor output color handling

* Fix pprint test

* fix flake8

Co-authored-by: Adrián Chaves <adrian@chaves.io>
Co-authored-by: Eugenio Lacuesta <eugenio.lacuesta@gmail.com>
 2020-07-20 17:23:38 +05:00
BroodingKangaroo 3e0492741d Another try to fix test errors on Windows 2020-07-19 00:10:29 +03:00
BroodingKangaroo 86f7ac2f2b Try to fix error at Windows 2020-07-17 18:22:47 +03:00
Artur Shellunts 62a4ede5e9
Remove deprecated classes BaseSgmlLinkExtractor, RegexLinkExtractor and SgmlLinkExtractor (#4356) 2020-07-17 12:40:04 +02:00
Adrián Chaves d29bec60d7
Upgrade PyPy for CI, and test both 3.5 (oldest) and 3.6 (newest) (#4504) 
* Upgrade PyPy for CI, and test both 3.5 (oldest) and 3.6 (newest)

* Log a detailed error message to discover why MockServer is not working

* Go for all lines!

* Disable tests based on mitmproxy while running on PyPy

* Fix test_get_func_args for PyPy 3.6+

* Make testPayloadDefaultCiphers work regardless of OpenSSL default ciphers

* Crossing fingers…

* Rename: testPayloadDefaultCiphers → testPayloadDisabledCipher

* Test the PyPy version currently documented as the minimum required version

* Fix the PYPY_VERSION tag

* Update the documentation about supported PyPy versions

* Also test the latest 3.5 Python version with PyPy

* Fix the PYPY_VERSION value for the latest 3.5 version

* Use pinned dependencies for asyncio and PyPy tests against oldest supported Python versions

* Fix PyPy installation for the pypy3-pinned Tox environment

* Try installing Cython

* Maybe PyPy requires lxml 3.6.0?

* install.rst: minor clarification

* lxml 4.0.0 is required on PyPy

* Require setuptools 18.5+

* Revert "Require setuptools 18.5+"

This reverts commit 017ec33ac2.

* Maintain lxml as a dependency if setuptools < 18.5 is used
 2020-07-17 02:19:24 +05:00
Mikhail Korobov 9a74a71c63
Merge pull request #4682 from noviluni/remove_python2_reminiscence 
remove python 2 reminiscence in cookies
 2020-07-16 23:58:58 +05:00
Adrián Chaves c4f9250220
Merge branch 'master' into ISSUE-4250-add_batch_deliveries 2020-07-16 17:44:18 +02:00
BroodingKangaroo 41263f61c6 Change single quotes to double in example in docs 2020-07-16 18:41:45 +03:00
Marc Hernández b97a39fda0
deprecate retry_on_eintr (#4683) 2020-07-16 17:38:22 +02:00
Mikhail Korobov 07470e1a3c
Merge pull request #3608 from ejulio/feat-685 
[MRG+1] Fix for #685 Add Google Cloud Storage Feed Export
 2020-07-16 18:11:19 +05:00
Mikhail Korobov 0f2f1acf04
Merge pull request #4458 from scrapy/azure-pipelines 
Set up CI with Azure Pipelines
 2020-07-16 18:09:31 +05:00
Marc 0e0d1ad643 remove python 2 reminiscence in cookies 2020-07-16 14:25:46 +02:00
BroodingKangaroo 8bdcdb0a76 Add quotes to example in docs 2020-07-16 09:13:54 +03:00
Júlio César Batista 38496a00b7
Use the itemlaoders library (#4516) 2020-07-15 12:08:36 +02:00
Andrey Rahmatullin eb1bc74417
Merge pull request #4674 from ashellunts/delete_deprecated_htmlparserlinkextractor 
Remove deprecated class HtmlParserLinkExtractor
 2020-07-15 12:51:42 +05:00
Aditya e662762e6a chore: Handle ConnectionTerminated event 2020-07-15 07:48:34 +05:30
Aditya 1dd27a92fa feat: Idle Timeout for H2Connection (240s) 2020-07-14 22:46:32 +05:30
Artur Shellunts ed5247ca4c Remove htmlparser.py from tests/ignore.txt 2020-07-14 18:06:11 +02:00
Adrián Chaves 6e119bd3e3
Merge branch 'master' into azure-pipelines 2020-07-14 11:26:19 +02:00
Aditya aeaeb7385b feat: assert negotiated protocol as h2 
- implement IHandshakeListener in H2ClientProtocol to know when
handshake is completed
- implement IProtocolNegotiationFactory in H2ClientFactory to provide
information about the acceptableProtols (h2) during NPN or ALPN protocol
 2020-07-14 03:55:14 +05:30
Adrián Chaves 544c1f6e39 Fix the issue 2020-07-13 16:30:34 +02:00