RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
8,246 Commits 49 Branches 132 Tags 141 MiB
Commit Graph

8246 Commits

Author SHA1 Message Date
Aditya 9fffb801ed feat: H2Agent, H2ConnectionPool base implementation 2020-07-22 14:52:49 +05:30
Aditya 3685e99cca test: http2 connection timeout 2020-07-22 14:47:20 +05:30
Aditya 316620b517 chore: pass spider as argument for request method 
- download_maxsize and download_warnsize can now be extracted from the
spider directly and passed to the stream
- remove `partial` flag from the response as per RFC 7540 - Section
8.1.2.6
 2020-07-22 13:57:49 +05:30
Aditya e662762e6a chore: Handle ConnectionTerminated event 2020-07-15 07:48:34 +05:30
Aditya 1dd27a92fa feat: Idle Timeout for H2Connection (240s) 2020-07-14 22:46:32 +05:30
Aditya aeaeb7385b feat: assert negotiated protocol as h2 
- implement IHandshakeListener in H2ClientProtocol to know when
handshake is completed
- implement IProtocolNegotiationFactory in H2ClientFactory to provide
information about the acceptableProtols (h2) during NPN or ALPN protocol
 2020-07-14 03:55:14 +05:30
Aditya 64c6af10e1 refactor: use str instead of to_unicode 2020-07-13 00:57:49 +05:30
Aditya 2ea7d82534 feat: H2ClientFactory 2020-07-08 18:57:13 +05:30
Aditya 1c40dfa740 fix: handle CONNECTION_LOST & RESET separately 2020-07-07 15:36:41 +05:30
Eugenio Lacuesta 54e4228c3a
refactor: use protocol 
- H2ClientProtocol.close_stream
- Fix and add missing type hints
- More adjustments
- Rename stream id generator
- Simplify decrement
 2020-07-06 22:40:45 +05:30
Aditya 7f5bb6b34c chore: add h2 to setup.py, tox.ini 
- Change log level for hpack to ERROR
 2020-07-06 13:12:04 +05:30
Aditya a94b30342a test: reduce test data size to 1MB 2020-07-06 12:49:12 +05:30
Aditya 4acdc2e5d6 refactor: use __qualname__, () for large strings 2020-07-01 20:15:33 +05:30
Aditya c361fe0d3b feat: check for invalid hostname 
- Initiating requests having hostname or (ip_address, port) different
from the peer to which HTTP/2 connection is made can lead to closing the
whole connection and close out all the pending streams.
- This change aims to fix that problem
- Add required tests
- Save hostname & port in H2ConnectionMetadataDict
 2020-07-01 18:14:44 +05:30
Aditya 7b1ad995a4 test: query params, certificate & ip_address 
- refactor from str.format() to f-strings
 2020-07-01 10:45:36 +05:30
Aditya 50dd9271b4 fix: disable redundant logs 
- while testing the job exceeded the maximum log length
and was terminated
- reduce the number of requests from 20 to 10
 2020-06-30 07:36:56 +05:30
Aditya 26ab3e4137 feat: FIFO policy to handle large no. of requests 
- add required test -- test by sending 1000 requests
- increase test timeout to 180 seconds to account for tests taking long
time
 2020-06-30 06:44:20 +05:30
Aditya d17417b03a Merge branch 'master' of https://github.com/scrapy/scrapy into h2-client-protocol 2020-06-29 18:43:42 +05:30
Aditya Kumar e46b47c365
Renew the localhost certificate for tests (#4650) 
Validity
  Not Before: Jun 28 12:54:15 2020 GMT
  Not After : Jun 28 12:54:15 2021 GMT
Subject: C = IE, O = Scrapy, CN = localhost
 2020-06-29 15:05:13 +02:00
Aditya 90a7007f88 test: warnsize logs, no content header, dataloss 2020-06-29 18:29:31 +05:30
Aditya 23906b6bee refactor: move TypedDict types to types.py 
- rename LOGGER -> logger
- remove self._write_to_transport from Stream class and handle all
transport related activities inside HTTP2ClientProtocol class
 2020-06-29 18:21:05 +05:30
Marc Hernández 464f24f8c1
Add --data-raw to utils.curl and fix missing method with data (#4612) 2020-06-29 14:20:29 +02:00
Aditya 6387445ef5 test(tox.ini): change Twisted -> Twisted[http2] 2020-06-28 18:44:57 +05:30
Aditya 690dd7f38b test: GET & POST request test for h2 client 
- Remove repeated dependency Twisted from setup.py
- Test for both GET & POST when
  - Only 1 request
  - Large number (=20) of requests
and
  - Small Data (10 KB) per request
  - Large Data (10 MB) per request
- Test when request is cancelled by the client'

BREAKING CHANGES
Tests raises OpenSSL.SSL.Error when run using tox. However, all tests
passes when ran using `python -m unittest`.
 2020-06-28 16:38:49 +05:30
Mikhail Korobov 0c8d8c5e85
Merge pull request #4645 from Lukas0907/fix-starturlsspider 
Spider constructor expects name as argument, not start_urls.
 2020-06-25 18:36:43 +05:00
Adrián Chaves 9f60481360 Bump version: 2.1.0 → 2.2.0 2020-06-24 12:27:39 +02:00
Adrián Chaves 6f4c964aa4
Cover Scrapy 2.2.0 in the release notes (#4630) 2020-06-24 12:26:38 +02:00
Aditya 065b3153fe Merge branch 'master' of https://github.com/scrapy/scrapy into h2-client-protocol 2020-06-24 07:51:41 +05:30
Aditya 69f6d038c0 feat: TypedDict for Stream._response 
- remove test_protocol.py as working testing environment is setup 🙂🙃
- Add typing_extensions as dependency to support TypedDict for
python<3.8
 2020-06-24 07:50:21 +05:30
Aditya a97ac0adf8 test: GET request for HTTP2Client using mockserver 2020-06-24 06:40:20 +05:30
Lukas Anzinger 3672f5f988 Spider constructor expects name as argument, not start_urls. 
Fixes #4644
 2020-06-23 14:56:12 +02:00
Andrey Rahmatullin 536643efde
Merge pull request #4629 from StasDeep/fix/duplicated-feed-logs 
Fix duplicated feed logs
 2020-06-23 15:33:48 +05:00
Andrey Rahmatullin 4bee73cfa7
Merge pull request #4637 from elacuesta/typing 
Typing: Tox env, CI job
 2020-06-23 15:31:37 +05:00
Andrey Rahmatullin 374d996093
Merge pull request #4642 from elacuesta/docs-loaders-dataclass 
Docs: add note about dataclass items and loaders
 2020-06-23 15:30:48 +05:00
Eugenio Lacuesta 73b6ce8cb5
Update docs about dataclass items and loaders 2020-06-22 14:13:37 -03:00
Eugenio Lacuesta 1335d9053e
Update docs/topics/loaders.rst 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-06-22 14:05:44 -03:00
Eugenio Lacuesta 3efea98e05
Docs: add note about dataclass items and loaders 2020-06-22 12:41:25 -03:00
Aditya c74ef660c7 feat: handle response for different reasons 
- Add StreamCloseReason enum
- Send response for different cases considering download_warnsize,
download_maxsize, fail_on_data_loss, connection lost, etc.
 2020-06-21 09:34:23 +05:30
Aditya 303485a9b4 fix(http2): POST request not sending large body 2020-06-21 00:33:34 +05:30
Stanislau Hluboki a4bfd5ab6f Fix duplicated feed logs 2020-06-19 17:51:56 +03:00
Eugenio Lacuesta 7babf359e0
Typing: Tox env, CI job 2020-06-18 13:56:07 -03:00
Devi Sandeep 5d54173187
Update docs on accessing callback arguments in errback (#4634) 2020-06-18 12:01:38 +02:00
Aditya 700df3eeb7 test: mockserver with h2 protocol for tests 
- add Twisted[http2] in setup.py requirements
- add test_protocol.py to test the current implementation

BREAKING CHANGES
test_download times out because of no protocol negotiated between
Mockserver and HTTP/2 client
 2020-06-17 21:02:14 +05:30
Aditya 089dbc75e7 chore: use deque for pending request pool 
- Use itertools.count to generate next stream_id

BREAKING CHANGES
When sending data/body more than the local flow control window -- no
window update occurs to send the remaining data frames. Hence, the
complete body is not send resulting in no response received.
 2020-06-17 20:57:03 +05:30
Stas Glubokiy 3d027fb578
Fix missing storage.store calls in FeedExporter.close_spider (#4626) 2020-06-17 17:08:14 +02:00
Andrey Rahmatullin 89e900e013
Merge pull request #4635 from cool-RR/patch-1 
Use chain.from_iterable in python.py
 2020-06-17 18:54:10 +05:00
Ram Rachum 214da8e523
Use chain.from_iterable in python.py 2020-06-17 13:50:54 +03:00
Aditya 01ad8b31ab refactor(http2): clean up 
- make separate function to parse http headers from Request instance
 2020-06-15 05:14:00 +05:30
Aditya de4a34365a fix: large data chunk not received 
Every data chunk received needs to be acknowledged to
- update the flow control window size
- get furthur data chunks from the server
 2020-06-14 22:43:36 +05:30
Eugenio Lacuesta ec98dabfab
Support for dataclass and attrs items (#3881) 2020-06-14 11:45:27 +02:00