RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
8,251 Commits 49 Branches 132 Tags 160 MiB
Commit Graph

8251 Commits

Author SHA1 Message Date
Andrey Rahmatullin 89e900e013
Merge pull request #4635 from cool-RR/patch-1 
Use chain.from_iterable in python.py
 2020-06-17 18:54:10 +05:00
Ram Rachum 214da8e523
Use chain.from_iterable in python.py 2020-06-17 13:50:54 +03:00
Aditya 01ad8b31ab refactor(http2): clean up 
- make separate function to parse http headers from Request instance
 2020-06-15 05:14:00 +05:30
Aditya de4a34365a fix: large data chunk not received 
Every data chunk received needs to be acknowledged to
- update the flow control window size
- get furthur data chunks from the server
 2020-06-14 22:43:36 +05:30
Eugenio Lacuesta ec98dabfab
Support for dataclass and attrs items (#3881) 2020-06-14 11:45:27 +02:00
Aditya d06bb12e35 refactor: move H2Connection instance to stream 
- Remove all wrapper funtions made such that stream can send header/data
to H2Connection as they were not necessary

BREAKING CHANGES
Looks like, for small set of response data the StreamEnded event is
emitted and everything works well -- tested for both GET & POST request.
Maybe some issue with window size and/or flow control as when the
response data needs to be broken into separate chunks -- not all chunks
are received everytime which leads to indefinite waiting for next data
chunk and the connection is lost due to timeout. 😥

Working on setting up testing environment now. After testing is setup
I'll debug the above bug furthur.
 2020-06-13 22:33:28 +05:30
Aditya d09ccf8d3b feat(http2): support for POST requests 
BREAKING CHANGES
- Request is sent successfully with its Response received as well.
However, the StreamEnded event is not received which do not fires the
response deferred
 2020-06-13 20:40:01 +05:30
Eugenio Lacuesta 8b549392f9
Bump minimum Python version to 3.5.2 (#4615) 2020-06-11 14:53:59 +02:00
Andrey Rahmatullin 092f6fdea1
Merge pull request #4604 from MMesch/fix-startproject-permissions 
give write access to template files after copying with startproject
 2020-06-10 11:58:16 +05:00
Adrián Chaves 76a2cbf0ff
Apply minor style changes 2020-06-09 21:30:19 +02:00
Aditya 9ff9caecad feat(http2): support for GET requests 2020-06-07 14:04:53 +05:30
Matthias Meschede 2df3b54c7d refactor 2020-06-05 09:29:05 +02:00
Matthias Meschede 0cabf406d5 set write permission to startproject folder 2020-06-04 10:39:40 +02:00
Andrey Rahmatullin 63929e75e3
Merge pull request #4552 from jay24rajput/common_commands 
Code sharing between crawl and runspider command
 2020-06-03 12:06:12 +05:00
Eugenio Lacuesta 91e505edfe
Return single element from coroutine callback (#4609) 2020-06-02 10:32:14 +02:00
Aditya 791292334e chore(http2): Stream class 2020-06-02 09:13:31 +05:30
Mikhail Korobov 1d2b692666
Merge pull request #4596 from sharmarohit/remove-unneeded-escape-sequence 
remove unneeded escape sequence
 2020-06-02 02:30:33 +05:00
Adrián Chaves c86a1035dd
Merge pull request #4563 from willbeaufoy/hoverxref 
Extend hoverxref_roles to custom crossrefs (#4495)
 2020-06-01 07:18:13 +02:00
Bulat Khabibullin 5cef927944
Implement TextResponse.json() (#4574) 2020-06-01 06:57:23 +02:00
Aditya Kumar 9408c77a1e feat(http2): IH2EventsHandler, http2 module 2020-05-31 18:08:51 +05:30
Mikhail Korobov 6aab3badfa
Merge pull request #4568 from elacuesta/remove-monkeypatches-flake8-e402 
Remove _monkeypatches, import-related Flake8 codes
 2020-05-30 00:07:50 +05:00
Mikhail Korobov 13215cfb71
Merge pull request #4559 from elacuesta/cancel-request-download 
Stop response download from signal handler
 2020-05-27 21:46:00 +05:00
Andrey Rahmatullin ff699eb1f8
Merge pull request #4599 from elacuesta/LocalWeakReferencedCache-getitem-fix 
Fix KeyError in LocalWeakReferencedCache
 2020-05-27 21:42:04 +05:00
Andrey Rahmatullin 7bf375098e
Merge pull request #2400 from elacuesta/keep_cookie_header 
[MRG+1] CookiesMiddleware: keep cookies from 'Cookie' request header, fix encoding
 2020-05-27 21:41:43 +05:00
Eugenio Lacuesta 44d5801b05
Fix KeyError in LocalWeakReferencedCache 2020-05-27 11:15:33 -03:00
Eugenio Lacuesta 492197e477
StopDownload: store response in the exception instead of the failure 2020-05-27 10:45:07 -03:00
Eugenio Lacuesta 634ad5ebca
Merge remote-tracking branch 'upstream/master' into cancel-request-download 2020-05-27 10:27:45 -03:00
Will Beaufoy 898bdd3feb Update docs README with build instructions using tox 
None of the existing commands built the docs properly for me (I had to
revert the changes in 901892d to docs/conf.py to get them to build
properly, and even then no tooltips displayed).

Building them with tox worked for me, but other developers say they
can still use the original method, so the docs now contain both.
 2020-05-26 19:09:23 +01:00
Mikhail Korobov b82a480e91
Merge pull request #4588 from altendky/patch-1 
Incompatible with pytest 5.4 and 5.4.1
 2020-05-26 01:38:53 +05:00
Rohit Sharma 0cc0e51ca3 remove uneeded escape sequence 
removed uneeded wscape sequences from method arguments in the docs
folder
 2020-05-24 13:38:44 -07:00
Kyle Altendorf 5b2af85289
Link to issue describing troublesome pytest versions 2020-05-22 15:09:58 -04:00
Mikhail Korobov 3e854a69de
Merge pull request #4593 from Gallaecio/4514 
Fix the description of the ScreenshotPipeline example item pipeline
 2020-05-22 23:12:58 +05:00
Adrián Chaves 08756cd87a Fix the description of the ScreenshotPipeline example item pipeline 2020-05-22 18:06:54 +02:00
Eugenio Lacuesta f6ed5edc31
CookiesMiddleware: keep cookies from 'Cookie' request header 2020-05-21 11:07:41 -03:00
Eugenio Lacuesta fbe915d9cb
StopDownload: check partial response contents 2020-05-20 20:20:30 -03:00
Eugenio Lacuesta 9514393bd1
Reduce amount of lines 2020-05-20 15:28:54 -03:00
Eugenio Lacuesta 276721a5dc
Stop response download 2020-05-20 14:46:26 -03:00
Mikhail Korobov 6f3e3411b1
Merge pull request #4587 from Gallaecio/4510 
logging.rst: remove unused, misleading import
 2020-05-20 22:23:42 +05:00
Kyle Altendorf eb8493f09c
Incompatible with pytest 5.4 and 5.4.1 2020-05-20 09:21:34 -04:00
Adrián Chaves f418c6d5f7 logging.rst: remove unused, misleading import 2020-05-19 20:04:47 +02:00
Mikhail Korobov 6e79216425
Merge pull request #4585 from nsirletti/fix-readthedocs-build 
Make doc downloadable on ReadTheDocs.org
 2020-05-19 20:45:20 +05:00
nsirletti c2a0cca0fe Suppress Sphinx warnings about MIME types other than .rst 2020-05-19 02:17:11 +02:00
nsirletti afb2e5011f Allow doc to be downloadable on readthedocs.org 2020-05-19 02:17:11 +02:00
Mikhail Korobov febe82a907
Merge pull request #4486 from ilias-ant/add-file-status-on-media-pipelines-file-info 
Add status (downloaded, uptodate) to files information
 2020-05-18 22:30:30 +05:00
Mikhail Korobov bcc40c4077
better deprecation warning for Response.body_as_unicode() (#4579) 2020-05-18 19:12:31 +02:00
Eugenio Lacuesta a22f97052e
Cleanup import in selector module 2020-05-15 21:18:26 -03:00
Eugenio Lacuesta a915af2e45
Remove monkeypatches module, E402 flake8 code 2020-05-15 20:41:51 -03:00
Mikhail Korobov 314adf6c38
Merge pull request #4237 from elacuesta/flake8-max-line-length 
Set flake8-max-line-length to 119
 2020-05-16 03:26:45 +05:00
Mikhail Korobov 14612fc3d3
Merge pull request #4534 from elacuesta/deprecate-baseitem 
Deprecate scrapy.item.BaseItem
 2020-05-16 03:24:06 +05:00
Jay Rajput 604fe33bad
Update scrapy/commands/__init__.py 
Changed typo in a comment for BaseRunSpiderCommand

Co-authored-by: Eugenio Lacuesta <1731933+elacuesta@users.noreply.github.com>
 2020-05-16 01:53:49 +05:30