RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
8,430 Commits 49 Branches 132 Tags 174 MiB
Commit Graph

8430 Commits

Author SHA1 Message Date
adityaa30 75fe3d1365 fix: increase timeout to 0.5 seconds 
- In Windows specifically the reactor was left unclean by the
  HostnameEndpoint due to the tearDown method of
  test_downloader_handlers.py::HttpTestCase due to
  which the following 2 tests were failing:
  1. test_timeout_download_from_spider_server_hangs
  2. test_timeout_download_from_spider_nodata_rcvd
- Increasing the timeout fixed the test (in local)
 2020-08-17 04:18:03 +05:30
Aditya 38d361792c fix: typing & pylint errors 
- Ignore typing check for http2 test files
 2020-08-16 17:55:16 +05:30
Aditya f9f008e935 test: add typing-extensions 2020-08-16 17:04:40 +05:30
Aditya d97cf973dd Merge branch 'master' of https://github.com/scrapy/scrapy into h2-client-protocol 2020-08-16 11:31:10 +05:30
Aditya af73f141b2 refactor: move all http2 tests in separate files 2020-08-16 11:26:32 +05:30
Andrey Rahmatullin acb3b4433a
Merge pull request #4724 from Gallaecio/feed-uri-params 
Document FEED_URI_PARAMS
 2020-08-14 14:47:55 +05:00
Mikhail Korobov 61653418ff
Merge pull request #4090 from Gallaecio/documentation-build 
Fix references to Python types in parameter type fields
 2020-08-14 00:58:06 +05:00
Mikhail Korobov cd0d3fd48d
Merge pull request #4705 from Gallaecio/deprecation-policy 
Cover our deprecation policy in the documentation
 2020-08-14 00:27:37 +05:00
Mikhail Korobov 1f44464a4a
Merge pull request #4723 from Gallaecio/windows 
test_utils_iterators.py: support Windows the right way
 2020-08-14 00:23:55 +05:00
Adrián Chaves 65e0abaea5 Document FEED_URI_PARAMS 2020-08-13 09:05:51 +02:00
Adrián Chaves 24ba5a71ac Maybe the problem is not in the code after all 2020-08-13 06:35:09 +02:00
Adrián Chaves 5f4df622a1 test_utils_iterators.py: support Windows the right way 2020-08-13 05:41:06 +02:00
Ajay Mittur 1c4b4cc6b0
Support defining file path based on item in media pipelines (#4686) 2020-08-11 14:12:44 +02:00
Adrián Chaves b2f4df5cb7 Merge remote-tracking branch 'upstream/master' into documentation-build 2020-08-11 13:16:14 +02:00
Adrián Chaves 90f85a2b9b Enable Travis CI 2020-08-11 10:20:30 +02:00
Aditya c67d6dea31 fix: H2 docs, NotImplementedError for H2 Tunnel 2020-08-11 04:39:41 +05:30
Andrey Rahmatullin 0cf1340c29
Merge pull request #4663 from ajaymittur28/scrapy-check-test 
Add Tests to `scrapy check` command
 2020-08-10 16:18:29 +05:00
Andrey Rahmatullin 702de0480b
Merge pull request #4564 from jacty/patch-1 
Unnecessary update when value is None
 2020-08-10 16:14:42 +05:00
Aditya e0c3019d90 fix: ScrapyProxyH2Agent 
- add required test cases

BREAKING CHANGES
Presently the tests (in test_downloader_handlers.py)
1. test_download_without_proxy
2. test_download_with_proxy_https_timeout

collide with each other when run together. However, if both of the tests
are ran individually then both pass.
 2020-08-09 16:19:35 +05:30
Andrey Rahmatullin cad84458ac
Merge pull request #4714 from linchiwei123/fix 
Remove duplicated install_requires
 2020-08-06 12:17:04 +05:00
linchiwei123 4dc09f09aa
Update setup.py 2020-08-05 22:23:19 +08:00
Andrey Rahmatullin 1f0722c87e
Merge pull request #4701 from kshitijcode/weakkeycache-cleanup-4684 
Code cleanup scrapy.utils.python.WeakKeyCache #4684
 2020-08-05 17:48:41 +05:00
Kshitij Sharma 983b7ddf2e aesthetic fixes 2020-08-05 16:13:52 +05:30
Kshitij Sharma b35d1f2b2c deleted tester.py 2020-08-05 09:14:04 +05:30
Kshitij Sharma 9d84289109 deprecated weakkeycache by specifying in __init__ 2020-08-05 09:11:59 +05:30
Marc Hernández 336f19f5cc
Change super syntax (#4707) 2020-08-04 20:42:01 +02:00
Adrián Chaves 4ee538e44b
Update unicode references from Python 2 times in the documentation (#4703) 2020-08-04 20:34:11 +02:00
Adrián Chaves 1278e76d90 Bump version: 2.2.0 → 2.3.0 2020-08-04 20:07:02 +02:00
Adrián Chaves 3600582f56
Cover Scrapy 2.2.1 and 2.3 in the release notes (#4708) 2020-08-04 20:05:56 +02:00
Mikhail Korobov 015b71d89f
Merge pull request #4704 from Gallaecio/python2-u-prefixes 
Remove the u prefix from strings
 2020-07-31 01:23:00 +05:00
Aditya d707f8b5d9 docs: mention H2DownloadHandler in settings.rst 2020-07-30 18:06:21 +05:30
Adrián Chaves 6f4ccec567 Cover our deprecation policy in the documentation 2020-07-30 14:03:14 +02:00
Adrián Chaves 890b2138a6 Remove the u prefix from strings 2020-07-30 13:39:30 +02:00
Aditya a3fecaf07f test: fix host-name H2DownloadHandler tests 2020-07-30 15:45:27 +05:30
Kshitij Sharma 49337bd2ae Code cleanup scrapy.utils.python.WeakKeyCache #4684 and fixing ci alerts 2020-07-30 12:25:21 +05:30
Kshitij Sharma 33ddc3d4f3 Trigger CI 2020-07-30 09:47:08 +05:30
Mikhail Korobov 5e2d1bd187
Merge pull request #4434 from BroodingKangaroo/ISSUE-4250-add_batch_deliveries 
Feed exports: add batch deliveries
 2020-07-29 23:03:28 +05:00
Kshitij Sharma 403bc7020a Code cleanup scrapy.utils.python.WeakKeyCache #4684 and fixing ci alerts 2020-07-29 18:05:33 +05:30
Aditya 19f2b4b53d refactor: AcceptableProtocolsContextFactory 
- rename H2WrappedContextFactory to AcceptableProtocolsContextFactory
- AcceptableProtocolsContextFactory accepts an argument
acceptable_protocols which can be used to override the context factory
priority list of protocols during ALPN or NPN
 2020-07-29 17:25:59 +05:30
Aditya e8342996f6 test: H2DownloadHandler 
Following tests are skipped as Content-Length header not matching the
data received is considered as a ProtocolError
- test_download_broken_content_cause_data_loss
- test_download_broken_chunked_content_cause_data_loss
- test_download_broken_content_allow_data_loss
- test_download_broken_chunked_content_allow_data_loss
- test_download_broken_content_allow_data_loss_via_setting
- test_download_broken_chunked_content_allow_data_loss_via_setting

BREAKING CHANGES
The following tests currently fail
- test_content_length_zero_bodyless_post_request_headers
- test_host_header_seted_in_request_headers
- test_download_with_maxsize_very_large_file
 2020-07-29 13:51:01 +05:30
Aditya 92bec38591 feat: MethodNotAllowed405, Content-Length header 
- add tests to check for Content-Length header
- raise MethodNotAllowed405 when remote send 'HTTP/2.0 405 Method Not
Allowed'
 2020-07-29 13:49:14 +05:30
Kshitij Sharma e7a58fe157 Code cleanup scrapy.utils.python.WeakKeyCache #4684 2020-07-29 10:16:18 +05:30
Eugenio Lacuesta 5265853937
Use ItemAdapter.field_names when writing header in CsvItemExporter (#4668) 2020-07-28 11:15:14 +02:00
BroodingKangaroo a6c1d79b7c pep8 tiny changes 2020-07-28 11:53:05 +03:00
Mikhail Korobov ce0c25fc85
Merge pull request #4690 from elacuesta/typing-setup-remove-monkeypatches 
Remove monkeypatches module from mypy section in setup.cfg
 2020-07-23 17:56:08 +05:00
Eugenio Lacuesta 8fae3d5bb7
Remove monkeypatches module from mypy section in setup.cfg 2020-07-22 16:08:35 -03:00
Aditya 031bfc9c3b feat(wip): ScrapyH2Agent, ScrapyProxyH2Agent 2020-07-22 15:01:59 +05:30
Aditya 62ce842afc fix: multiple h2 connections to same uri 
- When multiple requests are sent to H2ConnectionPool to the same uri
while the connection is in connecting state -- multiple connections were
establised.
- Fixed the bug using a deque of all the request deferred's which fire
with the H2ClientProtocol (connection) instance when connection is
established
 2020-07-22 14:52:49 +05:30
Aditya 8252a6f8d8 fix: H2Agent not able to connect via SSL 
- add H2WrappedContextFactory class which wraps the context factory
passed to H2Agent and updates the SSL context acceptable protocols list
to only h2
 2020-07-22 14:52:49 +05:30
Aditya 9fffb801ed feat: H2Agent, H2ConnectionPool base implementation 2020-07-22 14:52:49 +05:30