RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
7,239 Commits 49 Branches 132 Tags 134 MiB
Commit Graph

957 Commits

Author SHA1 Message Date
OmarFarrag 0e8770a2f4 test for files pipeline ftp store 2019-09-06 15:47:57 +02:00
Eugenio Lacuesta 7e3a602d56
Suggested changes 2019-07-15 12:37:09 -03:00
Eugenio Lacuesta d5a2a70329
Fix import 2019-07-13 23:47:41 -03:00
Eugenio Lacuesta ef9a619214
More DeprecationWarnings 2019-07-13 23:36:39 -03:00
Eugenio Lacuesta eced544d64
Prevent even more DeprecationWarnings 2019-07-13 22:16:23 -03:00
Eugenio Lacuesta 40086dabb8 Prevent more DeprecationWarnings 2019-07-13 22:14:00 -03:00
Eugenio Lacuesta 92d624c161
[Tests] Prevent more TestCase DeprecationWarnings 
* assertRegexpMatches -> assertRegex
 2019-07-13 20:55:45 -03:00
Eugenio Lacuesta b714a372e2
[Tests] Prevent more DeprecationWarnings 
* assertRegexpMatches -> assertRegex
* invalid escape sequence \[
 2019-07-13 20:23:15 -03:00
Eugenio Lacuesta 09e27d2d2e
[Tests] Monkey patch unittest.TestCase to prevent DeprecationWarning(s) 2019-07-13 19:26:59 -03:00
Luiz Francisco Rodrigues da Silva df68c4b9b1 Add a setting to use active mode in FTPFeedStorage (#3829) 
Add a setting to use active mode in FTPFeedStorage

FTP servers can be configured in active and passive mode, by default
Python and Scrapy use passive mode and there was no way to use active.

This commit adds a setting FEED_STORAGE_FTP_ACTIVE to allow a feed
to be exported to a FTP server configured in active mode.
 2019-07-10 14:25:52 +02:00
nyov 5442c2d3c3 Updating S3FeedStorage instancing without AWS key. 2019-07-06 13:41:56 +00:00
nyov 7a398b7086 Deprecation removals for Scrapy 1.7 
Removing deprecations of 2015 and prior (pre-1.1)
 2019-07-06 13:41:56 +00:00
Daniel Graña 3d8f075b0a
Merge pull request #3840 from mabelvj/itemloader-errors 
[MRG+1] Itemloader errors #3836
 2019-07-05 08:55:43 -03:00
Mikhail Korobov 9aec7856b0
Merge pull request #3794 from csalazar/whitelist-form-methods-in-fromresponse 
[MRG+1] Fix form methods in FormRequest.from_response (#3777)
 2019-07-02 20:08:14 +05:00
Eugenio Lacuesta 312e57382c Request.cb_kwargs: update in spider middleware 2019-06-26 12:48:00 -03:00
Eugenio Lacuesta d4d68cf321 Request.cb_kwargs: update in downloader middleware 2019-06-26 12:31:41 -03:00
Eugenio Lacuesta 1f9f41b850 Move request.cb_kwargs tests to their own test file 2019-06-26 12:31:12 -03:00
Eugenio Lacuesta 428309ba1a Merge remote-tracking branch 'origin/master' into callback_kwargs 2019-06-26 11:03:31 -03:00
Mikhail Korobov c81d120bde
Merge pull request #3768 from Gallaecio/meta-ignore-tags 
Implement the METAREFRESH_IGNORE_TAGS setting
 2019-06-25 23:25:58 +03:00
Mabel Villalba b1e348b281 [itemloader-errors] updated Compose and MapCompose messages and added tests 
#3836
 2019-06-25 17:11:14 +02:00
Mabel Villalba a753ea7e16 [itemloader-errors] added test for Compose and MapCompose 
#3836
 2019-06-25 14:35:49 +02:00
Mabel Villalba 99eb03a810 [itemloader-errors] adapted compose test for the new error being returned by compose 
#3836
 2019-06-25 14:24:27 +02:00
Victor Torres f4f2b1695c Fix a memory leak on the Media Pipeline (Files and Images) (#3813) 
We're storing exceptions captured by Twisted on the media pipeline
cache, but we're also using the defer.returnValue method with our
own methods decorated with @defer.inlineCallbacks.

The defer.returnValue method passes returned values forward by
throwing a defer._DefGen_Return exception, which in its turn
extends the BaseException class and is captured by Twisted.

This way, the latest exception stored in the Failure's object may
also have an HtmlResponse object in its __context__ attribute. As
the Response object also keeps track of the Request object that
has originated it, you could figure it out how many RAM we're
wasting here.

This could easily lead to a Memory Leak problem when running
spiders with Media Pipeline enabled and a particular Request set
that tends to raise a significant number of exceptions.

Example triggers:
- media requests with 404 status responses
- user land exceptins coming from custom middlewares
- etc.
 2019-06-24 12:38:05 +02:00
Daniel Graña b53ff59a22
Merge pull request #3790 from andrewbaxter/master 
[MRG+1] Account for mangling when serializing requests with private callbacks
 2019-06-17 12:33:21 -03:00
Daniel Graña 31cdb940ff
Merge pull request #3833 from anubhavp28/remove_unused_var 
removes unused var
 2019-06-17 11:50:49 -03:00
Anubhav Patel b8900ec6a6 removes unused var 2019-06-17 00:06:44 +05:30
Daniel Graña cdfb20aee8
Merge pull request #3739 from Matthijsy/feature/scrapy_check_env 
[MRG+1] Add SCRAPY_CHECK environment variable
 2019-06-13 19:43:01 -03:00
sortafreel cdeccac6d6 Linting (return previous indentation). 2019-06-11 17:38:06 +03:00
sortafreel 7dad2f7b13 Add more tests. 2019-06-11 07:43:03 +03:00
sortafreel a1bca6a8e7 Add tests. 2019-06-11 07:36:29 +03:00
Claudio Salazar 0c50879568 Change behavior to use method GET when there are unknown methods 
in the form
 2019-06-06 22:10:59 +02:00
Andrew Baxter 6af1dc89aa Fix mangling test 2019-06-06 04:25:19 +09:00
Andrew Baxter 3dd3e8c298 Restrict different class mangling tests to Py 3+ 2019-06-05 23:49:54 +09:00
Andrew Baxter 9c81721c40 Add tests for private method name mangling 2019-06-05 23:43:56 +09:00
Andrew Baxter bcad8947e8 Support inherited private method names 2019-06-03 20:41:02 +09:00
Andrew Baxter 9af91a26b0 Replace regex usage 2019-05-28 01:40:26 +09:00
Andrew Baxter 72b7d3e90a Make the regex align to the spec better; add unit tests for name variations 2019-05-27 23:30:23 +09:00
Claudio Salazar 461682fc3d Whitelist form methods in FormRequest.from_response method 2019-05-25 11:01:19 +02:00
Andrew Baxter e667ca7682 Account for mangling when serializing requests with private callbacks 2019-05-24 21:45:53 +09:00
Adrián Chaves 611249bb7f Implement the METAREFRESH_IGNORE_TAGS setting 2019-05-08 12:52:29 +02:00
Matthijs Vos f6485e6697 Restore alphabetic order and two lines between import and code 2019-05-03 13:55:18 +02:00
Matthijs Vos 6d52708579 Add reset case 2019-04-18 15:19:23 +02:00
Matthijs Vos 2973998947 Add set_environ test 2019-04-18 14:50:02 +02:00
Mikhail Korobov 5b667b6156
Merge pull request #3707 from Gallaecio/pytest-xdist 
Use pytest-xdist
 2019-04-17 13:20:37 +05:00
Adrián Chaves 5a6fb3daa6 Use pytest-xdist 2019-04-17 09:02:34 +02:00
Mikhail Korobov 9280185054
Merge pull request #3520 from whalebot-helmsman/round-robin-scheduler-tested 
[MRG+1] Downloader-aware Priority Queue for Scrapy
 2019-04-07 00:06:08 +05:00
Maram Sumanth aa46e1995c [MRG+1] Show elapsed time in statscollector (#3638) 
* Update corestats.py

* Update corestats.py

* corrected tests

* Update corestats.py

* Update scrapy/extensions/corestats.py
 2019-04-07 00:03:40 +05:00
Kostis Anagnostopoulos 3a493b6066 fix: do not catch system exceptions like KeyboardInterrupt 2019-04-05 11:52:00 +02:00
Konstantin Lopuhin b5c552d17f
Merge pull request #2061 from elacuesta/process_spider_exception_generator 
[MRG+1] process_spider_exception on generators
 2019-04-01 10:43:19 +03:00
Eugenio Lacuesta f5e0b6b89a parse command: rename cb_kwargs option to cbkwargs 2019-03-29 14:03:26 -03:00