RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,393 Commits 49 Branches 132 Tags 143 MiB
Commit Graph

10393 Commits

Author SHA1 Message Date
Lukas Anzinger 3672f5f988 Spider constructor expects name as argument, not start_urls. 
Fixes #4644
 2020-06-23 14:56:12 +02:00
Andrey Rahmatullin 536643efde
Merge pull request #4629 from StasDeep/fix/duplicated-feed-logs 
Fix duplicated feed logs
 2020-06-23 15:33:48 +05:00
Andrey Rahmatullin 4bee73cfa7
Merge pull request #4637 from elacuesta/typing 
Typing: Tox env, CI job
 2020-06-23 15:31:37 +05:00
Andrey Rahmatullin 374d996093
Merge pull request #4642 from elacuesta/docs-loaders-dataclass 
Docs: add note about dataclass items and loaders
 2020-06-23 15:30:48 +05:00
Adrián Chaves cfd039aeb6 Remove a duplicate GCS_PROJECT_ID reference target 2020-06-22 19:28:33 +02:00
Eugenio Lacuesta 73b6ce8cb5
Update docs about dataclass items and loaders 2020-06-22 14:13:37 -03:00
Eugenio Lacuesta 1335d9053e
Update docs/topics/loaders.rst 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-06-22 14:05:44 -03:00
Eugenio Lacuesta 3efea98e05
Docs: add note about dataclass items and loaders 2020-06-22 12:41:25 -03:00
Aditya c74ef660c7 feat: handle response for different reasons 
- Add StreamCloseReason enum
- Send response for different cases considering download_warnsize,
download_maxsize, fail_on_data_loss, connection lost, etc.
 2020-06-21 09:34:23 +05:30
Aditya 303485a9b4 fix(http2): POST request not sending large body 2020-06-21 00:33:34 +05:30
Eugenio Lacuesta b99fe4aa4c
Add google-cloud-storage to the 'pinned' tox environment 2020-06-19 21:41:15 -03:00
Eugenio Lacuesta c7f1c7e3f3
Merge branch 'master' into feat-685 2020-06-19 21:30:54 -03:00
Stanislau Hluboki a4bfd5ab6f Fix duplicated feed logs 2020-06-19 17:51:56 +03:00
Eugenio Lacuesta 7babf359e0
Typing: Tox env, CI job 2020-06-18 13:56:07 -03:00
Adrián Chaves f9c63384c0
Merge branch 'master' into allow-customizing-export-column-names 2020-06-18 14:09:20 +02:00
Adrián Chaves 88ade19675
Merge branch 'master' into spider-name-collision 2020-06-18 14:01:02 +02:00
Devi Sandeep 5d54173187
Update docs on accessing callback arguments in errback (#4634) 2020-06-18 12:01:38 +02:00
Aditya 700df3eeb7 test: mockserver with h2 protocol for tests 
- add Twisted[http2] in setup.py requirements
- add test_protocol.py to test the current implementation

BREAKING CHANGES
test_download times out because of no protocol negotiated between
Mockserver and HTTP/2 client
 2020-06-17 21:02:14 +05:30
Aditya 089dbc75e7 chore: use deque for pending request pool 
- Use itertools.count to generate next stream_id

BREAKING CHANGES
When sending data/body more than the local flow control window -- no
window update occurs to send the remaining data frames. Hence, the
complete body is not send resulting in no response received.
 2020-06-17 20:57:03 +05:30
Stas Glubokiy 3d027fb578
Fix missing storage.store calls in FeedExporter.close_spider (#4626) 2020-06-17 17:08:14 +02:00
Andrey Rahmatullin 89e900e013
Merge pull request #4635 from cool-RR/patch-1 
Use chain.from_iterable in python.py
 2020-06-17 18:54:10 +05:00
Ram Rachum 214da8e523
Use chain.from_iterable in python.py 2020-06-17 13:50:54 +03:00
Aditya 01ad8b31ab refactor(http2): clean up 
- make separate function to parse http headers from Request instance
 2020-06-15 05:14:00 +05:30
Aditya de4a34365a fix: large data chunk not received 
Every data chunk received needs to be acknowledged to
- update the flow control window size
- get furthur data chunks from the server
 2020-06-14 22:43:36 +05:30
Eugenio Lacuesta ec98dabfab
Support for dataclass and attrs items (#3881) 2020-06-14 11:45:27 +02:00
Aditya d06bb12e35 refactor: move H2Connection instance to stream 
- Remove all wrapper funtions made such that stream can send header/data
to H2Connection as they were not necessary

BREAKING CHANGES
Looks like, for small set of response data the StreamEnded event is
emitted and everything works well -- tested for both GET & POST request.
Maybe some issue with window size and/or flow control as when the
response data needs to be broken into separate chunks -- not all chunks
are received everytime which leads to indefinite waiting for next data
chunk and the connection is lost due to timeout. 😥

Working on setting up testing environment now. After testing is setup
I'll debug the above bug furthur.
 2020-06-13 22:33:28 +05:30
Aditya d09ccf8d3b feat(http2): support for POST requests 
BREAKING CHANGES
- Request is sent successfully with its Response received as well.
However, the StreamEnded event is not received which do not fires the
response deferred
 2020-06-13 20:40:01 +05:30
Eugenio Lacuesta 8b549392f9
Bump minimum Python version to 3.5.2 (#4615) 2020-06-11 14:53:59 +02:00
Eugenio Lacuesta b6c5289fb9
Close page in pyppeteer example, mention asyncio reactor 2020-06-10 12:11:49 -03:00
Andrey Rahmatullin 092f6fdea1
Merge pull request #4604 from MMesch/fix-startproject-permissions 
give write access to template files after copying with startproject
 2020-06-10 11:58:16 +05:00
Adrián Chaves 76a2cbf0ff
Apply minor style changes 2020-06-09 21:30:19 +02:00
Eugenio Lacuesta 78aa1b2bfc
Fix typo 2020-06-08 11:19:15 -03:00
Aditya 9ff9caecad feat(http2): support for GET requests 2020-06-07 14:04:53 +05:30
Eugenio Lacuesta bdabc500aa
Update headless browser docs 2020-06-06 16:47:15 -03:00
Matthias Meschede 2df3b54c7d refactor 2020-06-05 09:29:05 +02:00
Matthias Meschede 0cabf406d5 set write permission to startproject folder 2020-06-04 10:39:40 +02:00
Andrey Rahmatullin 63929e75e3
Merge pull request #4552 from jay24rajput/common_commands 
Code sharing between crawl and runspider command
 2020-06-03 12:06:12 +05:00
Eugenio Lacuesta 91e505edfe
Return single element from coroutine callback (#4609) 2020-06-02 10:32:14 +02:00
Aditya 791292334e chore(http2): Stream class 2020-06-02 09:13:31 +05:30
Mikhail Korobov 1d2b692666
Merge pull request #4596 from sharmarohit/remove-unneeded-escape-sequence 
remove unneeded escape sequence
 2020-06-02 02:30:33 +05:00
Adrián Chaves c86a1035dd
Merge pull request #4563 from willbeaufoy/hoverxref 
Extend hoverxref_roles to custom crossrefs (#4495)
 2020-06-01 07:18:13 +02:00
Bulat Khabibullin 5cef927944
Implement TextResponse.json() (#4574) 2020-06-01 06:57:23 +02:00
Aditya Kumar 9408c77a1e feat(http2): IH2EventsHandler, http2 module 2020-05-31 18:08:51 +05:30
Mikhail Korobov 6aab3badfa
Merge pull request #4568 from elacuesta/remove-monkeypatches-flake8-e402 
Remove _monkeypatches, import-related Flake8 codes
 2020-05-30 00:07:50 +05:00
Mikhail Korobov 13215cfb71
Merge pull request #4559 from elacuesta/cancel-request-download 
Stop response download from signal handler
 2020-05-27 21:46:00 +05:00
Andrey Rahmatullin ff699eb1f8
Merge pull request #4599 from elacuesta/LocalWeakReferencedCache-getitem-fix 
Fix KeyError in LocalWeakReferencedCache
 2020-05-27 21:42:04 +05:00
Andrey Rahmatullin 7bf375098e
Merge pull request #2400 from elacuesta/keep_cookie_header 
[MRG+1] CookiesMiddleware: keep cookies from 'Cookie' request header, fix encoding
 2020-05-27 21:41:43 +05:00
Eugenio Lacuesta 44d5801b05
Fix KeyError in LocalWeakReferencedCache 2020-05-27 11:15:33 -03:00
Eugenio Lacuesta 492197e477
StopDownload: store response in the exception instead of the failure 2020-05-27 10:45:07 -03:00
Eugenio Lacuesta 634ad5ebca
Merge remote-tracking branch 'upstream/master' into cancel-request-download 2020-05-27 10:27:45 -03:00