RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
7,236 Commits 49 Branches 132 Tags 182 MiB
Commit Graph

4401 Commits

Author SHA1 Message Date
OmarFarrag 0e8770a2f4 test for files pipeline ftp store 2019-09-06 15:47:57 +02:00
OmarFarrag 97d2f717ae Support extracting ftp settings in `ImagesPipeline` 2019-08-22 16:19:01 +02:00
OmarFarrag 2047124b35 Follow PEP8 .. Remove unnecessary comments 2019-08-22 16:18:14 +02:00
OmarFarrag bd22b25ef4 Make `stat_file` thread safe .. Refactor file storing.. Support act/psv 2019-08-22 01:30:15 +02:00
OmarFarrag 8c970c636e
port from str to int 
Co-Authored-By: Adrián Chaves <adrian@chaves.io>
 2019-08-21 18:28:36 +02:00
OmarFarrag 790bf90312 Make FTP persiting files thread safe 2019-08-19 19:16:47 +02:00
OmarFarrag 81ac1da381 Handle leading and trailing slashes 2019-08-19 17:17:21 +02:00
OmarFarrag 0a5cb7745b Fix reference mistake 2019-08-19 17:12:11 +02:00
OmarFarrag 9b1587ed1b Credentials from settings-Support custom paths-Remove close conenction 2019-08-19 16:13:56 +02:00
OmarFarrag c025003da2 Add FTPFileStore 2019-08-18 04:44:09 +02:00
Adrián Chaves 4e23d70dd3 Bump version: 1.6.0 → 1.7.0 2019-07-18 15:38:25 +02:00
Maram Sumanth 0d51f9cc27 [MRG+1] Wrong value of log_count/INFO in stats (#3643) 
* Update statscollectors.py

* Update statscollectors.py

* Update statscollectors.py

* Update crawler.py

* Update crawler.py

* corrected tests

* Update test_utils_log.py

* Update crawler.py

* Update crawler.py

* interchanged order

* correced
 2019-07-17 14:36:49 +05:00
Mikhail Korobov b2c013feca
Merge pull request #3878 from elacuesta/mergedict_to_chainmap 
Deprecate scrapy.utils.datatypes.MergeDict in favor of collections.ChainMap
 2019-07-17 14:31:50 +05:00
Eugenio Lacuesta d7074d86d2
Change condition to raise deprecation warning 2019-07-16 14:13:45 -03:00
Eugenio Lacuesta 7e3a602d56
Suggested changes 2019-07-15 12:37:09 -03:00
Eugenio Lacuesta ef9a619214
More DeprecationWarnings 2019-07-13 23:36:39 -03:00
Eugenio Lacuesta 62f3e22481
Deprecate scrapy.utils.datatypes.MergeDict in favor of collections.ChainMap 2019-07-13 22:51:38 -03:00
Eugenio Lacuesta eced544d64
Prevent even more DeprecationWarnings 2019-07-13 22:16:23 -03:00
Eugenio Lacuesta 40086dabb8 Prevent more DeprecationWarnings 2019-07-13 22:14:00 -03:00
Eugenio Lacuesta c24b80e1e6
Prevent DeprecationWarning about logging.warn 2019-07-13 20:34:31 -03:00
Eugenio Lacuesta 27e63e6890
Monkey patch configparser module to ease the dropping of py2 support 2019-07-13 20:10:06 -03:00
Luiz Francisco Rodrigues da Silva df68c4b9b1 Add a setting to use active mode in FTPFeedStorage (#3829) 
Add a setting to use active mode in FTPFeedStorage

FTP servers can be configured in active and passive mode, by default
Python and Scrapy use passive mode and there was no way to use active.

This commit adds a setting FEED_STORAGE_FTP_ACTIVE to allow a feed
to be exported to a FTP server configured in active mode.
 2019-07-10 14:25:52 +02:00
Mikhail Korobov 3dff65ac2c
Merge pull request #3857 from mikolaje/master 
PEP8 in cmdline.py
 2019-07-08 13:54:35 +05:00
Adrián Chaves 9cab3f247d
Merge pull request #3859 from scrapy/more-deprecations 
More deprecations
 2019-07-08 09:46:48 +02:00
Mikhail Korobov cb4477db3e deprecate scrapy.utils.markup 
it was an import-only shim for w3lib.html
 2019-07-08 11:18:40 +05:00
Mikhail Korobov 64ff3cd6aa deprecate scrapy.utils.multipart 
It was a shim for w3lib.form, but w3lib.form is deprecated as well.
 2019-07-08 11:18:15 +05:00
Mikhail Korobov 1130711cc1 deprecate scrapy.utils.http 
ChunkedTransferMiddleware is deprecated, so decode_chunked_transfer
can be deprecated as well.
 2019-07-08 11:17:30 +05:00
nyov 5442c2d3c3 Updating S3FeedStorage instancing without AWS key. 2019-07-06 13:41:56 +00:00
nyov 7a398b7086 Deprecation removals for Scrapy 1.7 
Removing deprecations of 2015 and prior (pre-1.1)
 2019-07-06 13:41:56 +00:00
mikolaje 8f52dada55 PEP8 in cmdline.py 2019-07-06 19:55:09 +08:00
Daniel Graña 3d8f075b0a
Merge pull request #3840 from mabelvj/itemloader-errors 
[MRG+1] Itemloader errors #3836
 2019-07-05 08:55:43 -03:00
Mikhail Korobov 9aec7856b0
Merge pull request #3794 from csalazar/whitelist-form-methods-in-fromresponse 
[MRG+1] Fix form methods in FormRequest.from_response (#3777)
 2019-07-02 20:08:14 +05:00
Claudio Salazar 2e4dc20393 Add backward compability when method=None in FormRequest 2019-06-26 21:36:28 +02:00
Eugenio Lacuesta 428309ba1a Merge remote-tracking branch 'origin/master' into callback_kwargs 2019-06-26 11:03:31 -03:00
Mabel Villalba 268a37cb7b [itemloader-errors] updated Compose and MapCompose function names 
#3836
 2019-06-26 13:23:10 +02:00
Mikhail Korobov c81d120bde
Merge pull request #3768 from Gallaecio/meta-ignore-tags 
Implement the METAREFRESH_IGNORE_TAGS setting
 2019-06-25 23:25:58 +03:00
Adrián Chaves d7795484dc Document scrapy.crawler.Crawler.stop() 2019-06-25 17:29:49 +02:00
Mabel Villalba b1e348b281 [itemloader-errors] updated Compose and MapCompose messages and added tests 
#3836
 2019-06-25 17:11:14 +02:00
Mabel Villalba ef56e34a4a [itemloader-errors] fixed typo 
#3836
 2019-06-25 13:56:53 +02:00
Mabel Villalba ad8c980010 [itemloader-errors] undo _proc 
#3836
 2019-06-25 13:54:47 +02:00
Mabel Villalba f134b1daf3 [itemloader-errors] reordered method 
#3836
 2019-06-25 13:46:44 +02:00
Mabel Villalba e5d17b4efe [itemloader-errors] reordered method 
#3836
 2019-06-25 13:44:20 +02:00
Mabel Villalba 859008a10b [itemloader-errors] added error message to Compose and MapCompose 
Fixes issue #3836
 2019-06-25 13:39:24 +02:00
Victor Torres f4f2b1695c Fix a memory leak on the Media Pipeline (Files and Images) (#3813) 
We're storing exceptions captured by Twisted on the media pipeline
cache, but we're also using the defer.returnValue method with our
own methods decorated with @defer.inlineCallbacks.

The defer.returnValue method passes returned values forward by
throwing a defer._DefGen_Return exception, which in its turn
extends the BaseException class and is captured by Twisted.

This way, the latest exception stored in the Failure's object may
also have an HtmlResponse object in its __context__ attribute. As
the Response object also keeps track of the Request object that
has originated it, you could figure it out how many RAM we're
wasting here.

This could easily lead to a Memory Leak problem when running
spiders with Media Pipeline enabled and a particular Request set
that tends to raise a significant number of exceptions.

Example triggers:
- media requests with 404 status responses
- user land exceptins coming from custom middlewares
- etc.
 2019-06-24 12:38:05 +02:00
Mabel Villalba 663352b2a5 [itemloader-errors] added error message to _process_input_value 2019-06-20 10:10:21 +02:00
Mabel Villalba 8d1e0e09bb [itemloader-errors] added error message in get_value 2019-06-20 10:06:06 +02:00
Daniel Graña b53ff59a22
Merge pull request #3790 from andrewbaxter/master 
[MRG+1] Account for mangling when serializing requests with private callbacks
 2019-06-17 12:33:21 -03:00
Daniel Graña cdfb20aee8
Merge pull request #3739 from Matthijsy/feature/scrapy_check_env 
[MRG+1] Add SCRAPY_CHECK environment variable
 2019-06-13 19:43:01 -03:00
Daniel Graña 296429669f
Merge pull request #3792 from barraponto/patch-4 
[MRG+1] Add 429 to RETRY_HTTP_CODES
 2019-06-13 17:17:08 -03:00
sortafreel a1bca6a8e7 Add tests. 2019-06-11 07:36:29 +03:00