RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
8,209 Commits 49 Branches 132 Tags 160 MiB
Commit Graph

1383 Commits

Author SHA1 Message Date
Andrey Rakhmatullin 351a400cf8 Merge remote-tracking branch 'origin/master' into asyncio-startrequests-asyncgen 2020-06-03 17:00:59 +05:00
Andrey Rakhmatullin 2f00a66237 Forbid running ExecutionEngine._next_request in parallel. 2020-06-03 13:52:51 +05:00
Eugenio Lacuesta 91e505edfe
Return single element from coroutine callback (#4609) 2020-06-02 10:32:14 +02:00
Bulat Khabibullin 5cef927944
Implement TextResponse.json() (#4574) 2020-06-01 06:57:23 +02:00
Mikhail Korobov 13215cfb71
Merge pull request #4559 from elacuesta/cancel-request-download 
Stop response download from signal handler
 2020-05-27 21:46:00 +05:00
Andrey Rahmatullin ff699eb1f8
Merge pull request #4599 from elacuesta/LocalWeakReferencedCache-getitem-fix 
Fix KeyError in LocalWeakReferencedCache
 2020-05-27 21:42:04 +05:00
Andrey Rahmatullin 7bf375098e
Merge pull request #2400 from elacuesta/keep_cookie_header 
[MRG+1] CookiesMiddleware: keep cookies from 'Cookie' request header, fix encoding
 2020-05-27 21:41:43 +05:00
Eugenio Lacuesta 44d5801b05
Fix KeyError in LocalWeakReferencedCache 2020-05-27 11:15:33 -03:00
Eugenio Lacuesta 492197e477
StopDownload: store response in the exception instead of the failure 2020-05-27 10:45:07 -03:00
Eugenio Lacuesta 634ad5ebca
Merge remote-tracking branch 'upstream/master' into cancel-request-download 2020-05-27 10:27:45 -03:00
Kyle Altendorf 5b2af85289
Link to issue describing troublesome pytest versions 2020-05-22 15:09:58 -04:00
Andrey Rakhmatullin 548281960b Remove process_iterable_helper from this PR. 2020-05-22 20:13:40 +05:00
Eugenio Lacuesta f6ed5edc31
CookiesMiddleware: keep cookies from 'Cookie' request header 2020-05-21 11:07:41 -03:00
Andrey Rakhmatullin 3bacf8e421 Fix test_utils_middlewares for 3.5. 2020-05-21 16:09:03 +05:00
Andrey Rakhmatullin 5b4adffabf Remove a duplicate test. 2020-05-21 15:23:47 +05:00
Andrey Rakhmatullin 8bf5f8915e Make test_utils_asyncgen actually run. 2020-05-21 15:20:05 +05:00
Andrey Rakhmatullin 0620895b7f Add simple tests for process_iterable_helper. 2020-05-21 15:17:22 +05:00
Eugenio Lacuesta fbe915d9cb
StopDownload: check partial response contents 2020-05-20 20:20:30 -03:00
Eugenio Lacuesta 276721a5dc
Stop response download 2020-05-20 14:46:26 -03:00
Kyle Altendorf eb8493f09c
Incompatible with pytest 5.4 and 5.4.1 2020-05-20 09:21:34 -04:00
Andrey Rakhmatullin 81483a7d98 Merge remote-tracking branch 'origin/master' into asyncio-startrequests-asyncgen 2020-05-19 11:43:10 +05:00
Mikhail Korobov febe82a907
Merge pull request #4486 from ilias-ant/add-file-status-on-media-pipelines-file-info 
Add status (downloaded, uptodate) to files information
 2020-05-18 22:30:30 +05:00
Mikhail Korobov 314adf6c38
Merge pull request #4237 from elacuesta/flake8-max-line-length 
Set flake8-max-line-length to 119
 2020-05-16 03:26:45 +05:00
Mikhail Korobov 14612fc3d3
Merge pull request #4534 from elacuesta/deprecate-baseitem 
Deprecate scrapy.item.BaseItem
 2020-05-16 03:24:06 +05:00
Adrián Chaves 36c3c9713e
Run tests with Python 3.5.0 (#4518) 
* Run tests with Python 3.5.0

* Use Ubuntu Trusty 14.04 to test Python 3.5.0

* Use mitmproxy<3.0.0 on Python 3.5.0

* Skip tests requiring mitmproxy in Python 3.5.0

* Change the minimum Python version from 3.5 to 3.5.1

* Do not prevent Scrapy from working with Python 3.5.0

* Force system Python 3.5.1

* Do not install a non-system Python in Python 3.5.1 jobs

* Switch to Trusty to be able to test Xenial’s Python version ¯\_(ツ)_/¯

* Add missing trusty

* Stop breaking old PyPy

* Allow installing Scrapy on Python 3.5.0
 2020-05-15 22:37:56 +05:00
Eugenio Lacuesta 3eeecb4273
Merge branch 'master' into flake8-max-line-length 2020-05-15 11:11:20 -03:00
Marc Hernández fffb0a5b6a
replace set constructor with set literals (#4573) 2020-05-15 15:23:09 +02:00
Eugenio Lacuesta ee13c3e95f
Merge branch 'master' into deprecate-baseitem 2020-05-14 13:31:16 -03:00
Eugenio Lacuesta 89d0c98f7a
Merge remote-tracking branch 'upstream/master' into flake8-remove-e128 2020-05-14 11:35:30 -03:00
Eugenio Lacuesta df8a1d1c01
Flake8: Remove E741 2020-05-14 10:40:50 -03:00
Adrián Chaves e31b6ccc45
Merge branch 'master' into flake8-remove-e128 2020-05-13 22:39:45 +02:00
Adrián Chaves 7dac99bb12
Merge branch 'master' into deprecate-baseitem 2020-05-13 20:35:03 +02:00
Andrey Rahmatullin 519f752d6d
Merge pull request #4566 from elacuesta/flake8-remove-f841 
Flake8: remove F841
 2020-05-13 11:46:48 +05:00
Aditya Kumar 97532a9144
test(spiderloader): no duplicate spider names (#4560) 2020-05-12 17:10:09 +02:00
willbeaufoy cf9be5344a
Prevent create_instance() returning None (#4532) 
Currently create_instance() can return None if an extension is
incorrectly implemented, but the extension will still show up as
enabled in the logs. This can cause confusion, as in the linked bug.

This change prevents this occurring by throwing an error if
create_instance() will return None.
 2020-05-11 20:35:25 +02:00
nsirletti cb8140a42a
Deprecate Response.body_as_unicode() (#4555) 
Co-authored-by: Nicolas Sirletti <n.sirletti@gmail.com>
 2020-05-11 20:20:31 +02:00
Eugenio Lacuesta 6f8758624c
Flake8: remove F841 2020-05-11 13:50:34 -03:00
Eugenio Lacuesta 25e9bc2d0d
Merge branch 'master' into flake8-remove-e128 2020-05-11 13:23:57 -03:00
Eugenio Lacuesta fb7ba6964d
Merge branch 'master' into deprecate-baseitem 2020-05-11 10:21:56 -03:00
Mikhail Korobov 892467cb8a
Merge pull request #4541 from elacuesta/pickle-adjustments 
Pickle: use protocol 4, update tests
 2020-05-11 15:16:14 +05:00
Mikhail Korobov b183579564
Merge pull request #4205 from elacuesta/bytes_received_signal 
Add bytes_received signal
 2020-05-11 15:09:55 +05:00
Eugenio Lacuesta 9485724d98
Merge branch 'master' into deprecate-baseitem 2020-05-08 19:32:47 -03:00
Eugenio Lacuesta be39f274b8
Merge branch 'master' into flake8-remove-e128 2020-05-08 18:33:29 -03:00
Mikhail Korobov 333910f6c8
Merge pull request #4554 from elacuesta/linkextractor-remove-lambdas 
Remove lambda attributes in default link extractor
 2020-05-09 02:11:59 +05:00
Eugenio Lacuesta c2c3054ac1
Flake8: remove E121 2020-05-08 16:43:47 -03:00
Eugenio Lacuesta 3ebf2a0d82
Remove lambdas in link extractor 2020-05-08 15:24:37 -03:00
Andrey Rakhmatullin c058156c48 Better idea for universal process_start_requests(). 2020-05-08 19:03:51 +05:00
Andrey Rakhmatullin 908682497d Merge remote-tracking branch 'origin/master' into asyncio-startrequests-asyncgen 2020-05-08 19:03:41 +05:00
Antonio Gordillo Toledo cf09af787e
Remove Python 2 encoding header from files (#4553) 2020-05-08 15:45:19 +02:00
Eugenio Lacuesta 6160370927
Merge remote-tracking branch 'upstream/master' into deprecate-baseitem 2020-05-07 14:38:54 -03:00