RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
8,175 Commits 49 Branches 132 Tags 180 MiB
Commit Graph

8175 Commits

Author SHA1 Message Date
Aditya 089dbc75e7 chore: use deque for pending request pool 
- Use itertools.count to generate next stream_id

BREAKING CHANGES
When sending data/body more than the local flow control window -- no
window update occurs to send the remaining data frames. Hence, the
complete body is not send resulting in no response received.
 2020-06-17 20:57:03 +05:30
Aditya 01ad8b31ab refactor(http2): clean up 
- make separate function to parse http headers from Request instance
 2020-06-15 05:14:00 +05:30
Aditya de4a34365a fix: large data chunk not received 
Every data chunk received needs to be acknowledged to
- update the flow control window size
- get furthur data chunks from the server
 2020-06-14 22:43:36 +05:30
Aditya d06bb12e35 refactor: move H2Connection instance to stream 
- Remove all wrapper funtions made such that stream can send header/data
to H2Connection as they were not necessary

BREAKING CHANGES
Looks like, for small set of response data the StreamEnded event is
emitted and everything works well -- tested for both GET & POST request.
Maybe some issue with window size and/or flow control as when the
response data needs to be broken into separate chunks -- not all chunks
are received everytime which leads to indefinite waiting for next data
chunk and the connection is lost due to timeout. 😥

Working on setting up testing environment now. After testing is setup
I'll debug the above bug furthur.
 2020-06-13 22:33:28 +05:30
Aditya d09ccf8d3b feat(http2): support for POST requests 
BREAKING CHANGES
- Request is sent successfully with its Response received as well.
However, the StreamEnded event is not received which do not fires the
response deferred
 2020-06-13 20:40:01 +05:30
Aditya 9ff9caecad feat(http2): support for GET requests 2020-06-07 14:04:53 +05:30
Aditya 791292334e chore(http2): Stream class 2020-06-02 09:13:31 +05:30
Aditya Kumar 9408c77a1e feat(http2): IH2EventsHandler, http2 module 2020-05-31 18:08:51 +05:30
Mikhail Korobov 6aab3badfa
Merge pull request #4568 from elacuesta/remove-monkeypatches-flake8-e402 
Remove _monkeypatches, import-related Flake8 codes
 2020-05-30 00:07:50 +05:00
Mikhail Korobov 13215cfb71
Merge pull request #4559 from elacuesta/cancel-request-download 
Stop response download from signal handler
 2020-05-27 21:46:00 +05:00
Andrey Rahmatullin ff699eb1f8
Merge pull request #4599 from elacuesta/LocalWeakReferencedCache-getitem-fix 
Fix KeyError in LocalWeakReferencedCache
 2020-05-27 21:42:04 +05:00
Andrey Rahmatullin 7bf375098e
Merge pull request #2400 from elacuesta/keep_cookie_header 
[MRG+1] CookiesMiddleware: keep cookies from 'Cookie' request header, fix encoding
 2020-05-27 21:41:43 +05:00
Eugenio Lacuesta 44d5801b05
Fix KeyError in LocalWeakReferencedCache 2020-05-27 11:15:33 -03:00
Eugenio Lacuesta 492197e477
StopDownload: store response in the exception instead of the failure 2020-05-27 10:45:07 -03:00
Eugenio Lacuesta 634ad5ebca
Merge remote-tracking branch 'upstream/master' into cancel-request-download 2020-05-27 10:27:45 -03:00
Mikhail Korobov b82a480e91
Merge pull request #4588 from altendky/patch-1 
Incompatible with pytest 5.4 and 5.4.1
 2020-05-26 01:38:53 +05:00
Kyle Altendorf 5b2af85289
Link to issue describing troublesome pytest versions 2020-05-22 15:09:58 -04:00
Mikhail Korobov 3e854a69de
Merge pull request #4593 from Gallaecio/4514 
Fix the description of the ScreenshotPipeline example item pipeline
 2020-05-22 23:12:58 +05:00
Adrián Chaves 08756cd87a Fix the description of the ScreenshotPipeline example item pipeline 2020-05-22 18:06:54 +02:00
Eugenio Lacuesta f6ed5edc31
CookiesMiddleware: keep cookies from 'Cookie' request header 2020-05-21 11:07:41 -03:00
Eugenio Lacuesta fbe915d9cb
StopDownload: check partial response contents 2020-05-20 20:20:30 -03:00
Eugenio Lacuesta 9514393bd1
Reduce amount of lines 2020-05-20 15:28:54 -03:00
Eugenio Lacuesta 276721a5dc
Stop response download 2020-05-20 14:46:26 -03:00
Mikhail Korobov 6f3e3411b1
Merge pull request #4587 from Gallaecio/4510 
logging.rst: remove unused, misleading import
 2020-05-20 22:23:42 +05:00
Kyle Altendorf eb8493f09c
Incompatible with pytest 5.4 and 5.4.1 2020-05-20 09:21:34 -04:00
Adrián Chaves f418c6d5f7 logging.rst: remove unused, misleading import 2020-05-19 20:04:47 +02:00
Mikhail Korobov 6e79216425
Merge pull request #4585 from nsirletti/fix-readthedocs-build 
Make doc downloadable on ReadTheDocs.org
 2020-05-19 20:45:20 +05:00
nsirletti c2a0cca0fe Suppress Sphinx warnings about MIME types other than .rst 2020-05-19 02:17:11 +02:00
nsirletti afb2e5011f Allow doc to be downloadable on readthedocs.org 2020-05-19 02:17:11 +02:00
Mikhail Korobov febe82a907
Merge pull request #4486 from ilias-ant/add-file-status-on-media-pipelines-file-info 
Add status (downloaded, uptodate) to files information
 2020-05-18 22:30:30 +05:00
Mikhail Korobov bcc40c4077
better deprecation warning for Response.body_as_unicode() (#4579) 2020-05-18 19:12:31 +02:00
Eugenio Lacuesta a22f97052e
Cleanup import in selector module 2020-05-15 21:18:26 -03:00
Eugenio Lacuesta a915af2e45
Remove monkeypatches module, E402 flake8 code 2020-05-15 20:41:51 -03:00
Mikhail Korobov 314adf6c38
Merge pull request #4237 from elacuesta/flake8-max-line-length 
Set flake8-max-line-length to 119
 2020-05-16 03:26:45 +05:00
Mikhail Korobov 14612fc3d3
Merge pull request #4534 from elacuesta/deprecate-baseitem 
Deprecate scrapy.item.BaseItem
 2020-05-16 03:24:06 +05:00
Adrián Chaves 36c3c9713e
Run tests with Python 3.5.0 (#4518) 
* Run tests with Python 3.5.0

* Use Ubuntu Trusty 14.04 to test Python 3.5.0

* Use mitmproxy<3.0.0 on Python 3.5.0

* Skip tests requiring mitmproxy in Python 3.5.0

* Change the minimum Python version from 3.5 to 3.5.1

* Do not prevent Scrapy from working with Python 3.5.0

* Force system Python 3.5.1

* Do not install a non-system Python in Python 3.5.1 jobs

* Switch to Trusty to be able to test Xenial’s Python version ¯\_(ツ)_/¯

* Add missing trusty

* Stop breaking old PyPy

* Allow installing Scrapy on Python 3.5.0
 2020-05-15 22:37:56 +05:00
Eugenio Lacuesta 3eeecb4273
Merge branch 'master' into flake8-max-line-length 2020-05-15 11:11:20 -03:00
Marc Hernández fffb0a5b6a
replace set constructor with set literals (#4573) 2020-05-15 15:23:09 +02:00
Eugenio Lacuesta ee13c3e95f
Merge branch 'master' into deprecate-baseitem 2020-05-14 13:31:16 -03:00
Eugenio Lacuesta 3d58e5e3be
Merge pull request #4545 from elacuesta/flake8-remove-e128 
Flake8: Remove E128
 2020-05-14 13:28:50 -03:00
Eugenio Lacuesta 89d0c98f7a
Merge remote-tracking branch 'upstream/master' into flake8-remove-e128 2020-05-14 11:35:30 -03:00
Eugenio Lacuesta 602bb13bdc
Merge pull request #4558 from elacuesta/flake8-remove-e741 
Flake8: Remove E741
 2020-05-14 11:26:59 -03:00
Eugenio Lacuesta df8a1d1c01
Flake8: Remove E741 2020-05-14 10:40:50 -03:00
Eugenio Lacuesta cfe4bf7e50
Merge pull request #4572 from noviluni/add_missed_flake8_rules 
Flake8: remove E306 and F523
 2020-05-14 10:38:55 -03:00
Adrián Chaves e31b6ccc45
Merge branch 'master' into flake8-remove-e128 2020-05-13 22:39:45 +02:00
Adrián Chaves 7dac99bb12
Merge branch 'master' into deprecate-baseitem 2020-05-13 20:35:03 +02:00
marc 8971878c1d fix new detected flake8 cases 2020-05-13 14:11:10 +02:00
Andrey Rahmatullin 519f752d6d
Merge pull request #4566 from elacuesta/flake8-remove-f841 
Flake8: remove F841
 2020-05-13 11:46:48 +05:00
Aditya Kumar 8d1269bcbc
Cover chompjs in documentation (#4562) 2020-05-12 20:42:28 +02:00
Aditya Kumar 97532a9144
test(spiderloader): no duplicate spider names (#4560) 2020-05-12 17:10:09 +02:00