RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
7,163 Commits 49 Branches 132 Tags 162 MiB
Commit Graph

7163 Commits

Author SHA1 Message Date
Andrey Rakhmatullin 0de6ffc8e1 Fix super() call. 2019-07-11 13:12:56 +05:00
Andrey Rakhmatullin 0b9dce3a6c Add DOWNLOADER_CLIENT_TLS_VERBOSE_LOGGING setting. 2019-07-08 17:40:56 +05:00
Andrey Rakhmatullin 67a4000928 Work around older pyOpenSSL not having get_cipher_name or get_protocol_version_name. 2019-07-08 10:31:52 +05:00
Andrey Rakhmatullin 69b1d5d3d7 Log cipher, certificate and temp key info on establishing an SSL connection. 2019-07-04 12:13:39 +05:00
Mikhail Korobov 4c755f1ddc
Merge pull request #3692 from anubhavp28/doc-patch 
[MRG+1] doc for creating custom cache storage backend.
 2019-07-02 20:18:14 +05:00
Mikhail Korobov 9aec7856b0
Merge pull request #3794 from csalazar/whitelist-form-methods-in-fromresponse 
[MRG+1] Fix form methods in FormRequest.from_response (#3777)
 2019-07-02 20:08:14 +05:00
Claudio Salazar 2e4dc20393 Add backward compability when method=None in FormRequest 2019-06-26 21:36:28 +02:00
Daniel Graña 3adf09bc4f
Merge pull request #3563 from elacuesta/callback_kwargs 
[MRG+1] Callback kwargs
 2019-06-26 14:01:29 -03:00
Eugenio Lacuesta 312e57382c Request.cb_kwargs: update in spider middleware 2019-06-26 12:48:00 -03:00
Eugenio Lacuesta d4d68cf321 Request.cb_kwargs: update in downloader middleware 2019-06-26 12:31:41 -03:00
Eugenio Lacuesta 1f9f41b850 Move request.cb_kwargs tests to their own test file 2019-06-26 12:31:12 -03:00
Eugenio Lacuesta 428309ba1a Merge remote-tracking branch 'origin/master' into callback_kwargs 2019-06-26 11:03:31 -03:00
Mikhail Korobov c81d120bde
Merge pull request #3768 from Gallaecio/meta-ignore-tags 
Implement the METAREFRESH_IGNORE_TAGS setting
 2019-06-25 23:25:58 +03:00
Mikhail Korobov e5f12faf5f
Merge pull request #3703 from Gallaecio/ajax-docs 
Add a topic about reaching data that selectors cannot reach
 2019-06-25 23:11:59 +03:00
Mikhail Korobov bc137dd215
Merge pull request #3842 from Gallaecio/documentation-coverage 
Document scrapy.crawler.Crawler.stop()
 2019-06-25 22:28:55 +03:00
Adrián Chaves d7795484dc Document scrapy.crawler.Crawler.stop() 2019-06-25 17:29:49 +02:00
Adrián Chaves 8a3b15eb91 Document how to select dynamically-loaded content 2019-06-24 16:06:21 +02:00
Victor Torres f4f2b1695c Fix a memory leak on the Media Pipeline (Files and Images) (#3813) 
We're storing exceptions captured by Twisted on the media pipeline
cache, but we're also using the defer.returnValue method with our
own methods decorated with @defer.inlineCallbacks.

The defer.returnValue method passes returned values forward by
throwing a defer._DefGen_Return exception, which in its turn
extends the BaseException class and is captured by Twisted.

This way, the latest exception stored in the Failure's object may
also have an HtmlResponse object in its __context__ attribute. As
the Response object also keeps track of the Request object that
has originated it, you could figure it out how many RAM we're
wasting here.

This could easily lead to a Memory Leak problem when running
spiders with Media Pipeline enabled and a particular Request set
that tends to raise a significant number of exceptions.

Example triggers:
- media requests with 404 status responses
- user land exceptins coming from custom middlewares
- etc.
 2019-06-24 12:38:05 +02:00
Daniel Graña b53ff59a22
Merge pull request #3790 from andrewbaxter/master 
[MRG+1] Account for mangling when serializing requests with private callbacks
 2019-06-17 12:33:21 -03:00
Daniel Graña 31cdb940ff
Merge pull request #3833 from anubhavp28/remove_unused_var 
removes unused var
 2019-06-17 11:50:49 -03:00
Anubhav Patel b8900ec6a6 removes unused var 2019-06-17 00:06:44 +05:30
Daniel Graña cdfb20aee8
Merge pull request #3739 from Matthijsy/feature/scrapy_check_env 
[MRG+1] Add SCRAPY_CHECK environment variable
 2019-06-13 19:43:01 -03:00
Daniel Graña 296429669f
Merge pull request #3792 from barraponto/patch-4 
[MRG+1] Add 429 to RETRY_HTTP_CODES
 2019-06-13 17:17:08 -03:00
Daniel Graña 8a022ac6ea
Merge pull request #3810 from Gallaecio/documentation-coverage 
Skip scrapy.contracts private APIs in the documentation coverage report
 2019-06-13 17:16:07 -03:00
Daniel Graña c05d24c571
Merge pull request #3819 from sortafreel/fix_missing_values 
[MRG+1] Fix missing values
 2019-06-13 00:06:25 -03:00
Daniel Graña 66c40d0e5d
Merge pull request #3825 from Gallaecio/fix-py34-setup 
Require Twisted<=19.2.0 for Python 3.4
 2019-06-12 17:40:26 -03:00
Daniel Graña 1a6ce134a4
Merge pull request #3827 from Gallaecio/fix-appveyor-config 
Set the cloned directory as PYTHONPATH in appveyor.yml
 2019-06-12 17:40:09 -03:00
sortafreel cdeccac6d6 Linting (return previous indentation). 2019-06-11 17:38:06 +03:00
Adrián Chaves fe0f80f2f4 Set the cloned directory as PYTHONPATH in appveyor.yml 2019-06-11 15:56:27 +02:00
Adrián Chaves 0da972339b Require Twisted<=19.2.0 for Python 3.4 2019-06-11 14:11:38 +02:00
sortafreel 7dad2f7b13 Add more tests. 2019-06-11 07:43:03 +03:00
sortafreel a1bca6a8e7 Add tests. 2019-06-11 07:36:29 +03:00
Adrián Chaves c7ba72b5dc Skip scrapy.contracts private APIs in the documentation coverage report 2019-06-07 13:14:33 +02:00
Sortafreel 754f52b027 Preprocess values if item built from dict. 
https://github.com/scrapy/scrapy/issues/3804
 2019-06-07 03:20:45 +03:00
Sortafreel bd8a10384b Add values (if there're any) when initiating items from dicts 
https://github.com/scrapy/scrapy/issues/3804
 2019-06-07 01:50:03 +03:00
Claudio Salazar 0c50879568 Change behavior to use method GET when there are unknown methods 
in the form
 2019-06-06 22:10:59 +02:00
Andrew Baxter 6af1dc89aa Fix mangling test 2019-06-06 04:25:19 +09:00
Mikhail Korobov 29bbbaa4dd
Merge pull request #3812 from duketemon/master 
[MRG+1] Tutorial: scrapy shell example should say "text" not "title" (#3807)
 2019-06-05 20:00:26 +05:00
Andrew Baxter 3dd3e8c298 Restrict different class mangling tests to Py 3+ 2019-06-05 23:49:54 +09:00
Andrew Baxter 9c81721c40 Add tests for private method name mangling 2019-06-05 23:43:56 +09:00
Artem Kuchumov c81e15ed6e
Tutorial: scrapy shell example should say "text" not "title" (#3807) 
Tutorial: scrapy shell example should say "text" not "title"
 2019-06-05 13:15:23 +05:00
Mikhail Korobov f755507d68
Merge pull request #3567 from Gallaecio/documentation-coverage 
Add a Sphinx extension to generate documentation coverage information
 2019-06-04 18:47:24 +05:00
Mikhail Korobov a3dab41de8
Merge pull request #3806 from Gallaecio/statscollectors-currentmodule 
Fix a double indexing of the scrapy.statscollectors module in the doc…
 2019-06-04 18:11:08 +05:00
Adrián Chaves c7b5ad0e20 Add a Sphinx extension to generate documentation coverage information 2019-06-04 15:03:25 +02:00
Adrián Chaves ea209a0ea7 Fix module double indexing issues in the documentation 2019-06-04 14:35:27 +02:00
Andrew Baxter bcad8947e8 Support inherited private method names 2019-06-03 20:41:02 +09:00
Mikhail Korobov 890e7fbd5b
Merge pull request #3791 from barraponto/patch-3 
[MRG+1] Fix documentation for spiderloader
 2019-05-28 19:16:12 +01:00
Mikhail Korobov 8d50384210
Merge pull request #3797 from mar-heaven/master 
[MRG+1] remove a "is"
 2019-05-28 19:12:04 +01:00
Andrew Baxter 9af91a26b0 Replace regex usage 2019-05-28 01:40:26 +09:00
Andrew Baxter 72b7d3e90a Make the regex align to the spec better; add unit tests for name variations 2019-05-27 23:30:23 +09:00