RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
7,176 Commits 49 Branches 132 Tags 162 MiB
Commit Graph

945 Commits

Author SHA1 Message Date
Daniel Graña 3d8f075b0a
Merge pull request #3840 from mabelvj/itemloader-errors 
[MRG+1] Itemloader errors #3836
 2019-07-05 08:55:43 -03:00
Mikhail Korobov 9aec7856b0
Merge pull request #3794 from csalazar/whitelist-form-methods-in-fromresponse 
[MRG+1] Fix form methods in FormRequest.from_response (#3777)
 2019-07-02 20:08:14 +05:00
Eugenio Lacuesta 312e57382c Request.cb_kwargs: update in spider middleware 2019-06-26 12:48:00 -03:00
Eugenio Lacuesta d4d68cf321 Request.cb_kwargs: update in downloader middleware 2019-06-26 12:31:41 -03:00
Eugenio Lacuesta 1f9f41b850 Move request.cb_kwargs tests to their own test file 2019-06-26 12:31:12 -03:00
Eugenio Lacuesta 428309ba1a Merge remote-tracking branch 'origin/master' into callback_kwargs 2019-06-26 11:03:31 -03:00
Mikhail Korobov c81d120bde
Merge pull request #3768 from Gallaecio/meta-ignore-tags 
Implement the METAREFRESH_IGNORE_TAGS setting
 2019-06-25 23:25:58 +03:00
Mabel Villalba b1e348b281 [itemloader-errors] updated Compose and MapCompose messages and added tests 
#3836
 2019-06-25 17:11:14 +02:00
Mabel Villalba a753ea7e16 [itemloader-errors] added test for Compose and MapCompose 
#3836
 2019-06-25 14:35:49 +02:00
Mabel Villalba 99eb03a810 [itemloader-errors] adapted compose test for the new error being returned by compose 
#3836
 2019-06-25 14:24:27 +02:00
Victor Torres f4f2b1695c Fix a memory leak on the Media Pipeline (Files and Images) (#3813) 
We're storing exceptions captured by Twisted on the media pipeline
cache, but we're also using the defer.returnValue method with our
own methods decorated with @defer.inlineCallbacks.

The defer.returnValue method passes returned values forward by
throwing a defer._DefGen_Return exception, which in its turn
extends the BaseException class and is captured by Twisted.

This way, the latest exception stored in the Failure's object may
also have an HtmlResponse object in its __context__ attribute. As
the Response object also keeps track of the Request object that
has originated it, you could figure it out how many RAM we're
wasting here.

This could easily lead to a Memory Leak problem when running
spiders with Media Pipeline enabled and a particular Request set
that tends to raise a significant number of exceptions.

Example triggers:
- media requests with 404 status responses
- user land exceptins coming from custom middlewares
- etc.
 2019-06-24 12:38:05 +02:00
Daniel Graña b53ff59a22
Merge pull request #3790 from andrewbaxter/master 
[MRG+1] Account for mangling when serializing requests with private callbacks
 2019-06-17 12:33:21 -03:00
Daniel Graña 31cdb940ff
Merge pull request #3833 from anubhavp28/remove_unused_var 
removes unused var
 2019-06-17 11:50:49 -03:00
Anubhav Patel b8900ec6a6 removes unused var 2019-06-17 00:06:44 +05:30
Daniel Graña cdfb20aee8
Merge pull request #3739 from Matthijsy/feature/scrapy_check_env 
[MRG+1] Add SCRAPY_CHECK environment variable
 2019-06-13 19:43:01 -03:00
sortafreel cdeccac6d6 Linting (return previous indentation). 2019-06-11 17:38:06 +03:00
sortafreel 7dad2f7b13 Add more tests. 2019-06-11 07:43:03 +03:00
sortafreel a1bca6a8e7 Add tests. 2019-06-11 07:36:29 +03:00
Claudio Salazar 0c50879568 Change behavior to use method GET when there are unknown methods 
in the form
 2019-06-06 22:10:59 +02:00
Andrew Baxter 6af1dc89aa Fix mangling test 2019-06-06 04:25:19 +09:00
Andrew Baxter 3dd3e8c298 Restrict different class mangling tests to Py 3+ 2019-06-05 23:49:54 +09:00
Andrew Baxter 9c81721c40 Add tests for private method name mangling 2019-06-05 23:43:56 +09:00
Andrew Baxter bcad8947e8 Support inherited private method names 2019-06-03 20:41:02 +09:00
Andrew Baxter 9af91a26b0 Replace regex usage 2019-05-28 01:40:26 +09:00
Andrew Baxter 72b7d3e90a Make the regex align to the spec better; add unit tests for name variations 2019-05-27 23:30:23 +09:00
Claudio Salazar 461682fc3d Whitelist form methods in FormRequest.from_response method 2019-05-25 11:01:19 +02:00
Andrew Baxter e667ca7682 Account for mangling when serializing requests with private callbacks 2019-05-24 21:45:53 +09:00
Adrián Chaves 611249bb7f Implement the METAREFRESH_IGNORE_TAGS setting 2019-05-08 12:52:29 +02:00
Matthijs Vos f6485e6697 Restore alphabetic order and two lines between import and code 2019-05-03 13:55:18 +02:00
Matthijs Vos 6d52708579 Add reset case 2019-04-18 15:19:23 +02:00
Matthijs Vos 2973998947 Add set_environ test 2019-04-18 14:50:02 +02:00
Mikhail Korobov 5b667b6156
Merge pull request #3707 from Gallaecio/pytest-xdist 
Use pytest-xdist
 2019-04-17 13:20:37 +05:00
Adrián Chaves 5a6fb3daa6 Use pytest-xdist 2019-04-17 09:02:34 +02:00
Mikhail Korobov 9280185054
Merge pull request #3520 from whalebot-helmsman/round-robin-scheduler-tested 
[MRG+1] Downloader-aware Priority Queue for Scrapy
 2019-04-07 00:06:08 +05:00
Maram Sumanth aa46e1995c [MRG+1] Show elapsed time in statscollector (#3638) 
* Update corestats.py

* Update corestats.py

* corrected tests

* Update corestats.py

* Update scrapy/extensions/corestats.py
 2019-04-07 00:03:40 +05:00
Kostis Anagnostopoulos 3a493b6066 fix: do not catch system exceptions like KeyboardInterrupt 2019-04-05 11:52:00 +02:00
Konstantin Lopuhin b5c552d17f
Merge pull request #2061 from elacuesta/process_spider_exception_generator 
[MRG+1] process_spider_exception on generators
 2019-04-01 10:43:19 +03:00
Eugenio Lacuesta f5e0b6b89a parse command: rename cb_kwargs option to cbkwargs 2019-03-29 14:03:26 -03:00
Eugenio Lacuesta e8af6331b5 Add cb_kwargs option to the parse command 2019-03-28 15:03:40 -03:00
Eugenio Lacuesta 70a4d93aa3 Callback kwargs: more tests 2019-03-28 13:28:19 -03:00
Mikhail Korobov ec719f55e7
Merge pull request #3682 from elacuesta/rule_process_request_response_parameter 
[MRG+1] Rule.process_request: access Response object
 2019-03-28 01:39:54 +05:00
Eugenio Lacuesta c43a23152b Merge remote-tracking branch 'upstream/master' into callback_kwargs 2019-03-27 14:42:41 -03:00
Maram Sumanth 2fd8b7c28c [MRG+1] redirect_reasons in Request.meta (#3687) 2019-03-27 00:15:53 +05:00
Vostretsov Nikita 73e4ff5304 report warnings 2019-03-25 13:48:58 +00:00
Vostretsov Nikita 103fdc40c1 Merge remote-tracking branch 'upstream/master' into round-robin-scheduler-tested 2019-03-25 16:16:42 +03:00
Vostretsov Nikita 31b8a6b33a report warnings 2019-03-25 16:08:40 +03:00
Lucy Wang df574de8cc improve tests and fix some lint warnings (#6) 
* refactor downloader-aware test cases

* fix lint

* add doctest for _path_safe

* remove unused code

* better doctest
 2019-03-25 16:08:40 +03:00
Vostretsov Nikita 8afffb7234 Tests Cleanup 
add doctest for function
no need in this variables
move common assertion inside function
rename variable
rename variables
rename function
use function
this is not a method of public API
correct name for test
Update docs/topics/settings.rst

Co-Authored-By: whalebot-helmsman <whalebot.helmsman@gmail.com>
 2019-03-25 16:08:40 +03:00
Vostretsov Nikita 989bba6cb3 Revert "new signal" 
This reverts commit 646164fd7d6dd52061804d2df7424cff929bf739.
remove tests
Revert "emit new signal"

This reverts commit fcde0c6880678957a76af6083b6248f430a00fcf.
Revert "documentation for new signal"

This reverts commit 8aeb9f696ece95c16499a96767a7afa3d9c4abf4.
 2019-03-25 16:08:40 +03:00
Vostretsov Nikita 443fb98a47 Use downloader directly 
rename variable
remove old write function
remove unused imports
remove old read function
remove unused function
use mock methods
mock downloader
close downloader
add parse method
use new PQ class
create mock downloader
use downloader directly
remove mark/unmark mechanism
 2019-03-25 16:08:40 +03:00