RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
8,172 Commits 49 Branches 132 Tags 162 MiB
Commit Graph

8172 Commits

Author SHA1 Message Date
Aditya d06bb12e35 refactor: move H2Connection instance to stream 
- Remove all wrapper funtions made such that stream can send header/data
to H2Connection as they were not necessary

BREAKING CHANGES
Looks like, for small set of response data the StreamEnded event is
emitted and everything works well -- tested for both GET & POST request.
Maybe some issue with window size and/or flow control as when the
response data needs to be broken into separate chunks -- not all chunks
are received everytime which leads to indefinite waiting for next data
chunk and the connection is lost due to timeout. 😥

Working on setting up testing environment now. After testing is setup
I'll debug the above bug furthur.
 2020-06-13 22:33:28 +05:30
Aditya d09ccf8d3b feat(http2): support for POST requests 
BREAKING CHANGES
- Request is sent successfully with its Response received as well.
However, the StreamEnded event is not received which do not fires the
response deferred
 2020-06-13 20:40:01 +05:30
Aditya 9ff9caecad feat(http2): support for GET requests 2020-06-07 14:04:53 +05:30
Aditya 791292334e chore(http2): Stream class 2020-06-02 09:13:31 +05:30
Aditya Kumar 9408c77a1e feat(http2): IH2EventsHandler, http2 module 2020-05-31 18:08:51 +05:30
Mikhail Korobov 6aab3badfa
Merge pull request #4568 from elacuesta/remove-monkeypatches-flake8-e402 
Remove _monkeypatches, import-related Flake8 codes
 2020-05-30 00:07:50 +05:00
Mikhail Korobov 13215cfb71
Merge pull request #4559 from elacuesta/cancel-request-download 
Stop response download from signal handler
 2020-05-27 21:46:00 +05:00
Andrey Rahmatullin ff699eb1f8
Merge pull request #4599 from elacuesta/LocalWeakReferencedCache-getitem-fix 
Fix KeyError in LocalWeakReferencedCache
 2020-05-27 21:42:04 +05:00
Andrey Rahmatullin 7bf375098e
Merge pull request #2400 from elacuesta/keep_cookie_header 
[MRG+1] CookiesMiddleware: keep cookies from 'Cookie' request header, fix encoding
 2020-05-27 21:41:43 +05:00
Eugenio Lacuesta 44d5801b05
Fix KeyError in LocalWeakReferencedCache 2020-05-27 11:15:33 -03:00
Eugenio Lacuesta 492197e477
StopDownload: store response in the exception instead of the failure 2020-05-27 10:45:07 -03:00
Eugenio Lacuesta 634ad5ebca
Merge remote-tracking branch 'upstream/master' into cancel-request-download 2020-05-27 10:27:45 -03:00
Mikhail Korobov b82a480e91
Merge pull request #4588 from altendky/patch-1 
Incompatible with pytest 5.4 and 5.4.1
 2020-05-26 01:38:53 +05:00
Kyle Altendorf 5b2af85289
Link to issue describing troublesome pytest versions 2020-05-22 15:09:58 -04:00
Mikhail Korobov 3e854a69de
Merge pull request #4593 from Gallaecio/4514 
Fix the description of the ScreenshotPipeline example item pipeline
 2020-05-22 23:12:58 +05:00
Adrián Chaves 08756cd87a Fix the description of the ScreenshotPipeline example item pipeline 2020-05-22 18:06:54 +02:00
Eugenio Lacuesta f6ed5edc31
CookiesMiddleware: keep cookies from 'Cookie' request header 2020-05-21 11:07:41 -03:00
Eugenio Lacuesta fbe915d9cb
StopDownload: check partial response contents 2020-05-20 20:20:30 -03:00
Eugenio Lacuesta 9514393bd1
Reduce amount of lines 2020-05-20 15:28:54 -03:00
Eugenio Lacuesta 276721a5dc
Stop response download 2020-05-20 14:46:26 -03:00
Mikhail Korobov 6f3e3411b1
Merge pull request #4587 from Gallaecio/4510 
logging.rst: remove unused, misleading import
 2020-05-20 22:23:42 +05:00
Kyle Altendorf eb8493f09c
Incompatible with pytest 5.4 and 5.4.1 2020-05-20 09:21:34 -04:00
Adrián Chaves f418c6d5f7 logging.rst: remove unused, misleading import 2020-05-19 20:04:47 +02:00
Mikhail Korobov 6e79216425
Merge pull request #4585 from nsirletti/fix-readthedocs-build 
Make doc downloadable on ReadTheDocs.org
 2020-05-19 20:45:20 +05:00
nsirletti c2a0cca0fe Suppress Sphinx warnings about MIME types other than .rst 2020-05-19 02:17:11 +02:00
nsirletti afb2e5011f Allow doc to be downloadable on readthedocs.org 2020-05-19 02:17:11 +02:00
Mikhail Korobov febe82a907
Merge pull request #4486 from ilias-ant/add-file-status-on-media-pipelines-file-info 
Add status (downloaded, uptodate) to files information
 2020-05-18 22:30:30 +05:00
Mikhail Korobov bcc40c4077
better deprecation warning for Response.body_as_unicode() (#4579) 2020-05-18 19:12:31 +02:00
Eugenio Lacuesta a22f97052e
Cleanup import in selector module 2020-05-15 21:18:26 -03:00
Eugenio Lacuesta a915af2e45
Remove monkeypatches module, E402 flake8 code 2020-05-15 20:41:51 -03:00
Mikhail Korobov 314adf6c38
Merge pull request #4237 from elacuesta/flake8-max-line-length 
Set flake8-max-line-length to 119
 2020-05-16 03:26:45 +05:00
Mikhail Korobov 14612fc3d3
Merge pull request #4534 from elacuesta/deprecate-baseitem 
Deprecate scrapy.item.BaseItem
 2020-05-16 03:24:06 +05:00
Adrián Chaves 36c3c9713e
Run tests with Python 3.5.0 (#4518) 
* Run tests with Python 3.5.0

* Use Ubuntu Trusty 14.04 to test Python 3.5.0

* Use mitmproxy<3.0.0 on Python 3.5.0

* Skip tests requiring mitmproxy in Python 3.5.0

* Change the minimum Python version from 3.5 to 3.5.1

* Do not prevent Scrapy from working with Python 3.5.0

* Force system Python 3.5.1

* Do not install a non-system Python in Python 3.5.1 jobs

* Switch to Trusty to be able to test Xenial’s Python version ¯\_(ツ)_/¯

* Add missing trusty

* Stop breaking old PyPy

* Allow installing Scrapy on Python 3.5.0
 2020-05-15 22:37:56 +05:00
Eugenio Lacuesta 3eeecb4273
Merge branch 'master' into flake8-max-line-length 2020-05-15 11:11:20 -03:00
Marc Hernández fffb0a5b6a
replace set constructor with set literals (#4573) 2020-05-15 15:23:09 +02:00
Eugenio Lacuesta ee13c3e95f
Merge branch 'master' into deprecate-baseitem 2020-05-14 13:31:16 -03:00
Eugenio Lacuesta 3d58e5e3be
Merge pull request #4545 from elacuesta/flake8-remove-e128 
Flake8: Remove E128
 2020-05-14 13:28:50 -03:00
Eugenio Lacuesta 89d0c98f7a
Merge remote-tracking branch 'upstream/master' into flake8-remove-e128 2020-05-14 11:35:30 -03:00
Eugenio Lacuesta 602bb13bdc
Merge pull request #4558 from elacuesta/flake8-remove-e741 
Flake8: Remove E741
 2020-05-14 11:26:59 -03:00
Eugenio Lacuesta df8a1d1c01
Flake8: Remove E741 2020-05-14 10:40:50 -03:00
Eugenio Lacuesta cfe4bf7e50
Merge pull request #4572 from noviluni/add_missed_flake8_rules 
Flake8: remove E306 and F523
 2020-05-14 10:38:55 -03:00
Adrián Chaves e31b6ccc45
Merge branch 'master' into flake8-remove-e128 2020-05-13 22:39:45 +02:00
Adrián Chaves 7dac99bb12
Merge branch 'master' into deprecate-baseitem 2020-05-13 20:35:03 +02:00
marc 8971878c1d fix new detected flake8 cases 2020-05-13 14:11:10 +02:00
Andrey Rahmatullin 519f752d6d
Merge pull request #4566 from elacuesta/flake8-remove-f841 
Flake8: remove F841
 2020-05-13 11:46:48 +05:00
Aditya Kumar 8d1269bcbc
Cover chompjs in documentation (#4562) 2020-05-12 20:42:28 +02:00
Aditya Kumar 97532a9144
test(spiderloader): no duplicate spider names (#4560) 2020-05-12 17:10:09 +02:00
willbeaufoy cf9be5344a
Prevent create_instance() returning None (#4532) 
Currently create_instance() can return None if an extension is
incorrectly implemented, but the extension will still show up as
enabled in the logs. This can cause confusion, as in the linked bug.

This change prevents this occurring by throwing an error if
create_instance() will return None.
 2020-05-11 20:35:25 +02:00
nsirletti cb8140a42a
Deprecate Response.body_as_unicode() (#4555) 
Co-authored-by: Nicolas Sirletti <n.sirletti@gmail.com>
 2020-05-11 20:20:31 +02:00
Eugenio Lacuesta 6f8758624c
Flake8: remove F841 2020-05-11 13:50:34 -03:00