Commit Graph

8251 Commits

Author SHA1 Message Date
Aditya e8342996f6 test: H2DownloadHandler
Following tests are skipped as Content-Length header not matching the
data received is considered as a ProtocolError
- test_download_broken_content_cause_data_loss
- test_download_broken_chunked_content_cause_data_loss
- test_download_broken_content_allow_data_loss
- test_download_broken_chunked_content_allow_data_loss
- test_download_broken_content_allow_data_loss_via_setting
- test_download_broken_chunked_content_allow_data_loss_via_setting

BREAKING CHANGES
The following tests currently fail
- test_content_length_zero_bodyless_post_request_headers
- test_host_header_seted_in_request_headers
- test_download_with_maxsize_very_large_file
2020-07-29 13:51:01 +05:30
Aditya 92bec38591 feat: MethodNotAllowed405, Content-Length header
- add tests to check for Content-Length header
- raise MethodNotAllowed405 when remote send 'HTTP/2.0 405 Method Not
Allowed'
2020-07-29 13:49:14 +05:30
Aditya 031bfc9c3b feat(wip): ScrapyH2Agent, ScrapyProxyH2Agent 2020-07-22 15:01:59 +05:30
Aditya 62ce842afc fix: multiple h2 connections to same uri
- When multiple requests are sent to H2ConnectionPool to the same uri
while the connection is in connecting state -- multiple connections were
establised.
- Fixed the bug using a deque of all the request deferred's which fire
with the H2ClientProtocol (connection) instance when connection is
established
2020-07-22 14:52:49 +05:30
Aditya 8252a6f8d8 fix: H2Agent not able to connect via SSL
- add H2WrappedContextFactory class which wraps the context factory
passed to H2Agent and updates the SSL context acceptable protocols list
to only h2
2020-07-22 14:52:49 +05:30
Aditya 9fffb801ed feat: H2Agent, H2ConnectionPool base implementation 2020-07-22 14:52:49 +05:30
Aditya 3685e99cca test: http2 connection timeout 2020-07-22 14:47:20 +05:30
Aditya 316620b517 chore: pass spider as argument for request method
- download_maxsize and download_warnsize can now be extracted from the
spider directly and passed to the stream
- remove `partial` flag from the response as per RFC 7540 - Section
8.1.2.6
2020-07-22 13:57:49 +05:30
Aditya e662762e6a chore: Handle ConnectionTerminated event 2020-07-15 07:48:34 +05:30
Aditya 1dd27a92fa feat: Idle Timeout for H2Connection (240s) 2020-07-14 22:46:32 +05:30
Aditya aeaeb7385b feat: assert negotiated protocol as h2
- implement IHandshakeListener in H2ClientProtocol to know when
handshake is completed
- implement IProtocolNegotiationFactory in H2ClientFactory to provide
information about the acceptableProtols (h2) during NPN or ALPN protocol
2020-07-14 03:55:14 +05:30
Aditya 64c6af10e1 refactor: use str instead of to_unicode 2020-07-13 00:57:49 +05:30
Aditya 2ea7d82534 feat: H2ClientFactory 2020-07-08 18:57:13 +05:30
Aditya 1c40dfa740 fix: handle CONNECTION_LOST & RESET separately 2020-07-07 15:36:41 +05:30
Eugenio Lacuesta 54e4228c3a
refactor: use protocol
- H2ClientProtocol.close_stream
- Fix and add missing type hints
- More adjustments
- Rename stream id generator
- Simplify decrement
2020-07-06 22:40:45 +05:30
Aditya 7f5bb6b34c chore: add h2 to setup.py, tox.ini
- Change log level for hpack to ERROR
2020-07-06 13:12:04 +05:30
Aditya a94b30342a test: reduce test data size to 1MB 2020-07-06 12:49:12 +05:30
Aditya 4acdc2e5d6 refactor: use __qualname__, () for large strings 2020-07-01 20:15:33 +05:30
Aditya c361fe0d3b feat: check for invalid hostname
- Initiating requests having hostname or (ip_address, port) different
from the peer to which HTTP/2 connection is made can lead to closing the
whole connection and close out all the pending streams.
- This change aims to fix that problem
- Add required tests
- Save hostname & port in H2ConnectionMetadataDict
2020-07-01 18:14:44 +05:30
Aditya 7b1ad995a4 test: query params, certificate & ip_address
- refactor from str.format() to f-strings
2020-07-01 10:45:36 +05:30
Aditya 50dd9271b4 fix: disable redundant logs
- while testing the job exceeded the maximum log length
and was terminated
- reduce the number of requests from 20 to 10
2020-06-30 07:36:56 +05:30
Aditya 26ab3e4137 feat: FIFO policy to handle large no. of requests
- add required test -- test by sending 1000 requests
- increase test timeout to 180 seconds to account for tests taking long
time
2020-06-30 06:44:20 +05:30
Aditya d17417b03a Merge branch 'master' of https://github.com/scrapy/scrapy into h2-client-protocol 2020-06-29 18:43:42 +05:30
Aditya Kumar e46b47c365
Renew the localhost certificate for tests (#4650)
Validity
  Not Before: Jun 28 12:54:15 2020 GMT
  Not After : Jun 28 12:54:15 2021 GMT
Subject: C = IE, O = Scrapy, CN = localhost
2020-06-29 15:05:13 +02:00
Aditya 90a7007f88 test: warnsize logs, no content header, dataloss 2020-06-29 18:29:31 +05:30
Aditya 23906b6bee refactor: move TypedDict types to types.py
- rename LOGGER -> logger
- remove self._write_to_transport from Stream class and handle all
transport related activities inside HTTP2ClientProtocol class
2020-06-29 18:21:05 +05:30
Marc Hernández 464f24f8c1
Add --data-raw to utils.curl and fix missing method with data (#4612) 2020-06-29 14:20:29 +02:00
Aditya 6387445ef5 test(tox.ini): change Twisted -> Twisted[http2] 2020-06-28 18:44:57 +05:30
Aditya 690dd7f38b test: GET & POST request test for h2 client
- Remove repeated dependency Twisted from setup.py
- Test for both GET & POST when
  - Only 1 request
  - Large number (=20) of requests
and
  - Small Data (10 KB) per request
  - Large Data (10 MB) per request
- Test when request is cancelled by the client'

BREAKING CHANGES
Tests raises OpenSSL.SSL.Error when run using tox. However, all tests
passes when ran using `python -m unittest`.
2020-06-28 16:38:49 +05:30
Mikhail Korobov 0c8d8c5e85
Merge pull request #4645 from Lukas0907/fix-starturlsspider
Spider constructor expects name as argument, not start_urls.
2020-06-25 18:36:43 +05:00
Adrián Chaves 9f60481360 Bump version: 2.1.0 → 2.2.0 2020-06-24 12:27:39 +02:00
Adrián Chaves 6f4c964aa4
Cover Scrapy 2.2.0 in the release notes (#4630) 2020-06-24 12:26:38 +02:00
Aditya 065b3153fe Merge branch 'master' of https://github.com/scrapy/scrapy into h2-client-protocol 2020-06-24 07:51:41 +05:30
Aditya 69f6d038c0 feat: TypedDict for Stream._response
- remove test_protocol.py as working testing environment is setup 🙂🙃
- Add typing_extensions as dependency to support TypedDict for
python<3.8
2020-06-24 07:50:21 +05:30
Aditya a97ac0adf8 test: GET request for HTTP2Client using mockserver 2020-06-24 06:40:20 +05:30
Lukas Anzinger 3672f5f988 Spider constructor expects name as argument, not start_urls.
Fixes #4644
2020-06-23 14:56:12 +02:00
Andrey Rahmatullin 536643efde
Merge pull request #4629 from StasDeep/fix/duplicated-feed-logs
Fix duplicated feed logs
2020-06-23 15:33:48 +05:00
Andrey Rahmatullin 4bee73cfa7
Merge pull request #4637 from elacuesta/typing
Typing: Tox env, CI job
2020-06-23 15:31:37 +05:00
Andrey Rahmatullin 374d996093
Merge pull request #4642 from elacuesta/docs-loaders-dataclass
Docs: add note about dataclass items and loaders
2020-06-23 15:30:48 +05:00
Eugenio Lacuesta 73b6ce8cb5
Update docs about dataclass items and loaders 2020-06-22 14:13:37 -03:00
Eugenio Lacuesta 1335d9053e
Update docs/topics/loaders.rst
Co-authored-by: Adrián Chaves <adrian@chaves.io>
2020-06-22 14:05:44 -03:00
Eugenio Lacuesta 3efea98e05
Docs: add note about dataclass items and loaders 2020-06-22 12:41:25 -03:00
Aditya c74ef660c7 feat: handle response for different reasons
- Add StreamCloseReason enum
- Send response for different cases considering download_warnsize,
download_maxsize, fail_on_data_loss, connection lost, etc.
2020-06-21 09:34:23 +05:30
Aditya 303485a9b4 fix(http2): POST request not sending large body 2020-06-21 00:33:34 +05:30
Stanislau Hluboki a4bfd5ab6f Fix duplicated feed logs 2020-06-19 17:51:56 +03:00
Eugenio Lacuesta 7babf359e0
Typing: Tox env, CI job 2020-06-18 13:56:07 -03:00
Devi Sandeep 5d54173187
Update docs on accessing callback arguments in errback (#4634) 2020-06-18 12:01:38 +02:00
Aditya 700df3eeb7 test: mockserver with h2 protocol for tests
- add Twisted[http2] in setup.py requirements
- add test_protocol.py to test the current implementation

BREAKING CHANGES
test_download times out because of no protocol negotiated between
Mockserver and HTTP/2 client
2020-06-17 21:02:14 +05:30
Aditya 089dbc75e7 chore: use deque for pending request pool
- Use itertools.count to generate next stream_id

BREAKING CHANGES
When sending data/body more than the local flow control window -- no
window update occurs to send the remaining data frames. Hence, the
complete body is not send resulting in no response received.
2020-06-17 20:57:03 +05:30
Stas Glubokiy 3d027fb578
Fix missing storage.store calls in FeedExporter.close_spider (#4626) 2020-06-17 17:08:14 +02:00