RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
9,472 Commits 49 Branches 132 Tags 158 MiB
Commit Graph

9472 Commits

Author SHA1 Message Date
Mabel Villalba b1e348b281 [itemloader-errors] updated Compose and MapCompose messages and added tests 
#3836
 2019-06-25 17:11:14 +02:00
Mabel Villalba a753ea7e16 [itemloader-errors] added test for Compose and MapCompose 
#3836
 2019-06-25 14:35:49 +02:00
Mabel Villalba 99eb03a810 [itemloader-errors] adapted compose test for the new error being returned by compose 
#3836
 2019-06-25 14:24:27 +02:00
Mabel Villalba ef56e34a4a [itemloader-errors] fixed typo 
#3836
 2019-06-25 13:56:53 +02:00
Mabel Villalba ad8c980010 [itemloader-errors] undo _proc 
#3836
 2019-06-25 13:54:47 +02:00
Mabel Villalba f134b1daf3 [itemloader-errors] reordered method 
#3836
 2019-06-25 13:46:44 +02:00
Mabel Villalba e5d17b4efe [itemloader-errors] reordered method 
#3836
 2019-06-25 13:44:20 +02:00
Mabel Villalba 859008a10b [itemloader-errors] added error message to Compose and MapCompose 
Fixes issue #3836
 2019-06-25 13:39:24 +02:00
Adrián Chaves 8a3b15eb91 Document how to select dynamically-loaded content 2019-06-24 16:06:21 +02:00
Victor Torres f4f2b1695c Fix a memory leak on the Media Pipeline (Files and Images) (#3813) 
We're storing exceptions captured by Twisted on the media pipeline
cache, but we're also using the defer.returnValue method with our
own methods decorated with @defer.inlineCallbacks.

The defer.returnValue method passes returned values forward by
throwing a defer._DefGen_Return exception, which in its turn
extends the BaseException class and is captured by Twisted.

This way, the latest exception stored in the Failure's object may
also have an HtmlResponse object in its __context__ attribute. As
the Response object also keeps track of the Request object that
has originated it, you could figure it out how many RAM we're
wasting here.

This could easily lead to a Memory Leak problem when running
spiders with Media Pipeline enabled and a particular Request set
that tends to raise a significant number of exceptions.

Example triggers:
- media requests with 404 status responses
- user land exceptins coming from custom middlewares
- etc.
 2019-06-24 12:38:05 +02:00
Mabel Villalba 663352b2a5 [itemloader-errors] added error message to _process_input_value 2019-06-20 10:10:21 +02:00
Mabel Villalba 8d1e0e09bb [itemloader-errors] added error message in get_value 2019-06-20 10:06:06 +02:00
Daniel Graña b53ff59a22
Merge pull request #3790 from andrewbaxter/master 
[MRG+1] Account for mangling when serializing requests with private callbacks
 2019-06-17 12:33:21 -03:00
Daniel Graña 31cdb940ff
Merge pull request #3833 from anubhavp28/remove_unused_var 
removes unused var
 2019-06-17 11:50:49 -03:00
Anubhav Patel b8900ec6a6 removes unused var 2019-06-17 00:06:44 +05:30
Daniel Graña cdfb20aee8
Merge pull request #3739 from Matthijsy/feature/scrapy_check_env 
[MRG+1] Add SCRAPY_CHECK environment variable
 2019-06-13 19:43:01 -03:00
Daniel Graña 296429669f
Merge pull request #3792 from barraponto/patch-4 
[MRG+1] Add 429 to RETRY_HTTP_CODES
 2019-06-13 17:17:08 -03:00
Daniel Graña 8a022ac6ea
Merge pull request #3810 from Gallaecio/documentation-coverage 
Skip scrapy.contracts private APIs in the documentation coverage report
 2019-06-13 17:16:07 -03:00
Daniel Graña c05d24c571
Merge pull request #3819 from sortafreel/fix_missing_values 
[MRG+1] Fix missing values
 2019-06-13 00:06:25 -03:00
Daniel Graña 66c40d0e5d
Merge pull request #3825 from Gallaecio/fix-py34-setup 
Require Twisted<=19.2.0 for Python 3.4
 2019-06-12 17:40:26 -03:00
Daniel Graña 1a6ce134a4
Merge pull request #3827 from Gallaecio/fix-appveyor-config 
Set the cloned directory as PYTHONPATH in appveyor.yml
 2019-06-12 17:40:09 -03:00
sortafreel cdeccac6d6 Linting (return previous indentation). 2019-06-11 17:38:06 +03:00
Adrián Chaves fe0f80f2f4 Set the cloned directory as PYTHONPATH in appveyor.yml 2019-06-11 15:56:27 +02:00
Adrián Chaves 0da972339b Require Twisted<=19.2.0 for Python 3.4 2019-06-11 14:11:38 +02:00
sortafreel 7dad2f7b13 Add more tests. 2019-06-11 07:43:03 +03:00
sortafreel a1bca6a8e7 Add tests. 2019-06-11 07:36:29 +03:00
Adrián Chaves c7ba72b5dc Skip scrapy.contracts private APIs in the documentation coverage report 2019-06-07 13:14:33 +02:00
Sortafreel 754f52b027 Preprocess values if item built from dict. 
https://github.com/scrapy/scrapy/issues/3804
 2019-06-07 03:20:45 +03:00
Sortafreel bd8a10384b Add values (if there're any) when initiating items from dicts 
https://github.com/scrapy/scrapy/issues/3804
 2019-06-07 01:50:03 +03:00
Claudio Salazar 0c50879568 Change behavior to use method GET when there are unknown methods 
in the form
 2019-06-06 22:10:59 +02:00
Andrew Baxter 6af1dc89aa Fix mangling test 2019-06-06 04:25:19 +09:00
Mikhail Korobov 29bbbaa4dd
Merge pull request #3812 from duketemon/master 
[MRG+1] Tutorial: scrapy shell example should say "text" not "title" (#3807)
 2019-06-05 20:00:26 +05:00
Andrew Baxter 3dd3e8c298 Restrict different class mangling tests to Py 3+ 2019-06-05 23:49:54 +09:00
Andrew Baxter 9c81721c40 Add tests for private method name mangling 2019-06-05 23:43:56 +09:00
Artem Kuchumov c81e15ed6e
Tutorial: scrapy shell example should say "text" not "title" (#3807) 
Tutorial: scrapy shell example should say "text" not "title"
 2019-06-05 13:15:23 +05:00
Mikhail Korobov f755507d68
Merge pull request #3567 from Gallaecio/documentation-coverage 
Add a Sphinx extension to generate documentation coverage information
 2019-06-04 18:47:24 +05:00
Mikhail Korobov a3dab41de8
Merge pull request #3806 from Gallaecio/statscollectors-currentmodule 
Fix a double indexing of the scrapy.statscollectors module in the doc…
 2019-06-04 18:11:08 +05:00
Adrián Chaves c7b5ad0e20 Add a Sphinx extension to generate documentation coverage information 2019-06-04 15:03:25 +02:00
Adrián Chaves ea209a0ea7 Fix module double indexing issues in the documentation 2019-06-04 14:35:27 +02:00
Andrew Baxter bcad8947e8 Support inherited private method names 2019-06-03 20:41:02 +09:00
Mikhail Korobov 890e7fbd5b
Merge pull request #3791 from barraponto/patch-3 
[MRG+1] Fix documentation for spiderloader
 2019-05-28 19:16:12 +01:00
Mikhail Korobov 8d50384210
Merge pull request #3797 from mar-heaven/master 
[MRG+1] remove a "is"
 2019-05-28 19:12:04 +01:00
Andrew Baxter 9af91a26b0 Replace regex usage 2019-05-28 01:40:26 +09:00
Andrew Baxter 72b7d3e90a Make the regex align to the spec better; add unit tests for name variations 2019-05-27 23:30:23 +09:00
mar-heaven 18f01ea6ec
remove a "is" 
When I translated in Chinese, I found a needless "is"
 2019-05-27 17:15:30 +08:00
Anubhav Patel 90fdefcbca cache if convert_image has deprecated signature 2019-05-25 19:11:48 +05:30
Anubhav Patel da82ede8a0
describe method as a command 2019-05-25 17:19:10 +05:30
Claudio Salazar 461682fc3d Whitelist form methods in FormRequest.from_response method 2019-05-25 11:01:19 +02:00
Andrew Baxter 144afcee79 Use regex to check for private methods 2019-05-25 00:52:00 +09:00
Capi Etheriel 0ee2284fcc
Add 429 to RETRY_HTTP_CODES 2019-05-24 11:11:15 -03:00