RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
7,176 Commits 49 Branches 132 Tags 145 MiB
Commit Graph

7176 Commits

Author SHA1 Message Date
Mikhail Korobov cb4477db3e deprecate scrapy.utils.markup 
it was an import-only shim for w3lib.html
 2019-07-08 11:18:40 +05:00
Mikhail Korobov 64ff3cd6aa deprecate scrapy.utils.multipart 
It was a shim for w3lib.form, but w3lib.form is deprecated as well.
 2019-07-08 11:18:15 +05:00
Mikhail Korobov 1130711cc1 deprecate scrapy.utils.http 
ChunkedTransferMiddleware is deprecated, so decode_chunked_transfer
can be deprecated as well.
 2019-07-08 11:17:30 +05:00
Daniel Graña 3d8f075b0a
Merge pull request #3840 from mabelvj/itemloader-errors 
[MRG+1] Itemloader errors #3836
 2019-07-05 08:55:43 -03:00
Mikhail Korobov 4d4bd0e823
Merge pull request #3609 from Gallaecio/2253 
Document FilesPipeline.file_path and ImagesPipeline.file_path
 2019-07-04 22:05:46 +05:00
Mikhail Korobov 4c755f1ddc
Merge pull request #3692 from anubhavp28/doc-patch 
[MRG+1] doc for creating custom cache storage backend.
 2019-07-02 20:18:14 +05:00
Mikhail Korobov 9aec7856b0
Merge pull request #3794 from csalazar/whitelist-form-methods-in-fromresponse 
[MRG+1] Fix form methods in FormRequest.from_response (#3777)
 2019-07-02 20:08:14 +05:00
Claudio Salazar 2e4dc20393 Add backward compability when method=None in FormRequest 2019-06-26 21:36:28 +02:00
Daniel Graña 3adf09bc4f
Merge pull request #3563 from elacuesta/callback_kwargs 
[MRG+1] Callback kwargs
 2019-06-26 14:01:29 -03:00
Eugenio Lacuesta 312e57382c Request.cb_kwargs: update in spider middleware 2019-06-26 12:48:00 -03:00
Eugenio Lacuesta d4d68cf321 Request.cb_kwargs: update in downloader middleware 2019-06-26 12:31:41 -03:00
Eugenio Lacuesta 1f9f41b850 Move request.cb_kwargs tests to their own test file 2019-06-26 12:31:12 -03:00
Eugenio Lacuesta 428309ba1a Merge remote-tracking branch 'origin/master' into callback_kwargs 2019-06-26 11:03:31 -03:00
Mabel Villalba 268a37cb7b [itemloader-errors] updated Compose and MapCompose function names 
#3836
 2019-06-26 13:23:10 +02:00
Mikhail Korobov c81d120bde
Merge pull request #3768 from Gallaecio/meta-ignore-tags 
Implement the METAREFRESH_IGNORE_TAGS setting
 2019-06-25 23:25:58 +03:00
Mikhail Korobov e5f12faf5f
Merge pull request #3703 from Gallaecio/ajax-docs 
Add a topic about reaching data that selectors cannot reach
 2019-06-25 23:11:59 +03:00
Mikhail Korobov bc137dd215
Merge pull request #3842 from Gallaecio/documentation-coverage 
Document scrapy.crawler.Crawler.stop()
 2019-06-25 22:28:55 +03:00
Adrián Chaves d7795484dc Document scrapy.crawler.Crawler.stop() 2019-06-25 17:29:49 +02:00
Mabel Villalba b1e348b281 [itemloader-errors] updated Compose and MapCompose messages and added tests 
#3836
 2019-06-25 17:11:14 +02:00
Mabel Villalba a753ea7e16 [itemloader-errors] added test for Compose and MapCompose 
#3836
 2019-06-25 14:35:49 +02:00
Mabel Villalba 99eb03a810 [itemloader-errors] adapted compose test for the new error being returned by compose 
#3836
 2019-06-25 14:24:27 +02:00
Mabel Villalba ef56e34a4a [itemloader-errors] fixed typo 
#3836
 2019-06-25 13:56:53 +02:00
Mabel Villalba ad8c980010 [itemloader-errors] undo _proc 
#3836
 2019-06-25 13:54:47 +02:00
Mabel Villalba f134b1daf3 [itemloader-errors] reordered method 
#3836
 2019-06-25 13:46:44 +02:00
Mabel Villalba e5d17b4efe [itemloader-errors] reordered method 
#3836
 2019-06-25 13:44:20 +02:00
Mabel Villalba 859008a10b [itemloader-errors] added error message to Compose and MapCompose 
Fixes issue #3836
 2019-06-25 13:39:24 +02:00
Adrián Chaves 8a3b15eb91 Document how to select dynamically-loaded content 2019-06-24 16:06:21 +02:00
Victor Torres f4f2b1695c Fix a memory leak on the Media Pipeline (Files and Images) (#3813) 
We're storing exceptions captured by Twisted on the media pipeline
cache, but we're also using the defer.returnValue method with our
own methods decorated with @defer.inlineCallbacks.

The defer.returnValue method passes returned values forward by
throwing a defer._DefGen_Return exception, which in its turn
extends the BaseException class and is captured by Twisted.

This way, the latest exception stored in the Failure's object may
also have an HtmlResponse object in its __context__ attribute. As
the Response object also keeps track of the Request object that
has originated it, you could figure it out how many RAM we're
wasting here.

This could easily lead to a Memory Leak problem when running
spiders with Media Pipeline enabled and a particular Request set
that tends to raise a significant number of exceptions.

Example triggers:
- media requests with 404 status responses
- user land exceptins coming from custom middlewares
- etc.
 2019-06-24 12:38:05 +02:00
Mabel Villalba 663352b2a5 [itemloader-errors] added error message to _process_input_value 2019-06-20 10:10:21 +02:00
Mabel Villalba 8d1e0e09bb [itemloader-errors] added error message in get_value 2019-06-20 10:06:06 +02:00
Daniel Graña b53ff59a22
Merge pull request #3790 from andrewbaxter/master 
[MRG+1] Account for mangling when serializing requests with private callbacks
 2019-06-17 12:33:21 -03:00
Daniel Graña 31cdb940ff
Merge pull request #3833 from anubhavp28/remove_unused_var 
removes unused var
 2019-06-17 11:50:49 -03:00
Anubhav Patel b8900ec6a6 removes unused var 2019-06-17 00:06:44 +05:30
Daniel Graña cdfb20aee8
Merge pull request #3739 from Matthijsy/feature/scrapy_check_env 
[MRG+1] Add SCRAPY_CHECK environment variable
 2019-06-13 19:43:01 -03:00
Daniel Graña 296429669f
Merge pull request #3792 from barraponto/patch-4 
[MRG+1] Add 429 to RETRY_HTTP_CODES
 2019-06-13 17:17:08 -03:00
Daniel Graña 8a022ac6ea
Merge pull request #3810 from Gallaecio/documentation-coverage 
Skip scrapy.contracts private APIs in the documentation coverage report
 2019-06-13 17:16:07 -03:00
Daniel Graña c05d24c571
Merge pull request #3819 from sortafreel/fix_missing_values 
[MRG+1] Fix missing values
 2019-06-13 00:06:25 -03:00
Daniel Graña 66c40d0e5d
Merge pull request #3825 from Gallaecio/fix-py34-setup 
Require Twisted<=19.2.0 for Python 3.4
 2019-06-12 17:40:26 -03:00
Daniel Graña 1a6ce134a4
Merge pull request #3827 from Gallaecio/fix-appveyor-config 
Set the cloned directory as PYTHONPATH in appveyor.yml
 2019-06-12 17:40:09 -03:00
sortafreel cdeccac6d6 Linting (return previous indentation). 2019-06-11 17:38:06 +03:00
Adrián Chaves fe0f80f2f4 Set the cloned directory as PYTHONPATH in appveyor.yml 2019-06-11 15:56:27 +02:00
Adrián Chaves 0da972339b Require Twisted<=19.2.0 for Python 3.4 2019-06-11 14:11:38 +02:00
sortafreel 7dad2f7b13 Add more tests. 2019-06-11 07:43:03 +03:00
sortafreel a1bca6a8e7 Add tests. 2019-06-11 07:36:29 +03:00
Adrián Chaves c7ba72b5dc Skip scrapy.contracts private APIs in the documentation coverage report 2019-06-07 13:14:33 +02:00
Sortafreel 754f52b027 Preprocess values if item built from dict. 
https://github.com/scrapy/scrapy/issues/3804
 2019-06-07 03:20:45 +03:00
Sortafreel bd8a10384b Add values (if there're any) when initiating items from dicts 
https://github.com/scrapy/scrapy/issues/3804
 2019-06-07 01:50:03 +03:00
Claudio Salazar 0c50879568 Change behavior to use method GET when there are unknown methods 
in the form
 2019-06-06 22:10:59 +02:00
Andrew Baxter 6af1dc89aa Fix mangling test 2019-06-06 04:25:19 +09:00
Mikhail Korobov 29bbbaa4dd
Merge pull request #3812 from duketemon/master 
[MRG+1] Tutorial: scrapy shell example should say "text" not "title" (#3807)
 2019-06-05 20:00:26 +05:00