RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,710 Commits 49 Branches 132 Tags 155 MiB
Commit Graph

10710 Commits

Author SHA1 Message Date
Eugenio Lacuesta d5a2a70329
Fix import 2019-07-13 23:47:41 -03:00
Eugenio Lacuesta ef9a619214
More DeprecationWarnings 2019-07-13 23:36:39 -03:00
Eugenio Lacuesta 62f3e22481
Deprecate scrapy.utils.datatypes.MergeDict in favor of collections.ChainMap 2019-07-13 22:51:38 -03:00
Eugenio Lacuesta eced544d64
Prevent even more DeprecationWarnings 2019-07-13 22:16:23 -03:00
Eugenio Lacuesta 40086dabb8 Prevent more DeprecationWarnings 2019-07-13 22:14:00 -03:00
Eugenio Lacuesta 92d624c161
[Tests] Prevent more TestCase DeprecationWarnings 
* assertRegexpMatches -> assertRegex
 2019-07-13 20:55:45 -03:00
Eugenio Lacuesta c24b80e1e6
Prevent DeprecationWarning about logging.warn 2019-07-13 20:34:31 -03:00
Eugenio Lacuesta b714a372e2
[Tests] Prevent more DeprecationWarnings 
* assertRegexpMatches -> assertRegex
* invalid escape sequence \[
 2019-07-13 20:23:15 -03:00
Eugenio Lacuesta 27e63e6890
Monkey patch configparser module to ease the dropping of py2 support 2019-07-13 20:10:06 -03:00
Eugenio Lacuesta 09e27d2d2e
[Tests] Monkey patch unittest.TestCase to prevent DeprecationWarning(s) 2019-07-13 19:26:59 -03:00
Andrey Rakhmatullin a96a07bc76 Add a test for DOWNLOADER_CLIENT_TLS_VERBOSE_LOGGING. 2019-07-12 18:44:45 +05:00
Adrián Chaves fa6a0d799b
Merge pull request #3684 from Gallaecio/remove-ubuntu-topic 
Remove docs/topics/ubuntu.rst
 2019-07-12 14:03:11 +02:00
Adrián Chaves c4bcfb1aee
Merge pull request #3866 from Gallaecio/broad-crawl-memory-docs 
Extend the topic about broad crawls with information about memory usage
 2019-07-11 11:15:41 +02:00
Adrián Chaves ed487b7db3 broad-crawls.rst: Refactor the memory usage section 2019-07-11 11:14:22 +02:00
Andrey Rakhmatullin 98689b27a8 Improve the DOWNLOADER_CLIENTCONTEXTFACTORY doc. 2019-07-11 14:02:35 +05:00
Andrey Rakhmatullin 0de6ffc8e1 Fix super() call. 2019-07-11 13:12:56 +05:00
Robin 5cdf277017 Update broad-crawls.rst 
Added section on how to treat memory consumption problems of broad crawls.
 2019-07-11 08:49:24 +02:00
Luiz Francisco Rodrigues da Silva df68c4b9b1 Add a setting to use active mode in FTPFeedStorage (#3829) 
Add a setting to use active mode in FTPFeedStorage

FTP servers can be configured in active and passive mode, by default
Python and Scrapy use passive mode and there was no way to use active.

This commit adds a setting FEED_STORAGE_FTP_ACTIVE to allow a feed
to be exported to a FTP server configured in active mode.
 2019-07-10 14:25:52 +02:00
Mikhail Korobov 050a62a955
Merge pull request #3861 from Gallaecio/documentation-coverage 
Increase the documentation coverage of exceptions
 2019-07-10 10:18:34 +05:00
Adrián Chaves 68d898d493
Merge pull request #3712 from elacuesta/docs_crawlspider_link_text 
[Docs] CrawlSpider: update Rule docs
 2019-07-10 00:18:51 +02:00
Eugenio Lacuesta 9fb0f8454e
Extend docs about Crawling Rules 2019-07-09 15:30:22 -03:00
Eugenio Lacuesta d04e84c11d Merge remote-tracking branch 'upstream/master' into docs_crawlspider_link_text 2019-07-09 15:24:31 -03:00
Adrián Chaves a0bbbe8ffa Add the API documentation of ContractFail to the contracts topic 2019-07-09 12:33:57 +02:00
Adrián Chaves 578bccf3bb Skip scrapy.exceptions.UsageError in the documentation coverage report 2019-07-09 12:33:07 +02:00
Mikhail Korobov a97a9b57bf
Merge pull request #3621 from Gallaecio/bfo-for-few-requests 
Document that the crawl order is BFO for small numbers of start requests
 2019-07-08 21:26:04 +05:00
Mikhail Korobov dda12b6537
Merge pull request #3672 from Gallaecio/split-items 
Add a FAQ entry about splitting items in item pipelines
 2019-07-08 20:15:23 +05:00
Mikhail Korobov 488156b15f
Merge pull request #3860 from Gallaecio/documentation-coverage 
Skip scrapy.downloadermiddlewares private APIs in the documentation c…
 2019-07-08 20:13:55 +05:00
Adrián Chaves a0b09e0193 Skip scrapy.downloadermiddlewares private APIs in the documentation coverage report 2019-07-08 14:45:17 +02:00
Andrey Rakhmatullin 0b9dce3a6c Add DOWNLOADER_CLIENT_TLS_VERBOSE_LOGGING setting. 2019-07-08 17:40:56 +05:00
Adrián Chaves 783de8a8d8 Document how request concurrency settings impact request order 2019-07-08 14:23:21 +02:00
Adrián Chaves 6ea62cacc7 Add a FAQ entry about splitting items in item pipelines 2019-07-08 13:58:55 +02:00
Mikhail Korobov 3dff65ac2c
Merge pull request #3857 from mikolaje/master 
PEP8 in cmdline.py
 2019-07-08 13:54:35 +05:00
Adrián Chaves 9cab3f247d
Merge pull request #3859 from scrapy/more-deprecations 
More deprecations
 2019-07-08 09:46:48 +02:00
Mikhail Korobov 4bef6f438d
Merge pull request #3578 from nyov/deprecations 
Deprecation removals for Scrapy 1.7
 2019-07-08 11:32:07 +05:00
Mikhail Korobov cb4477db3e deprecate scrapy.utils.markup 
it was an import-only shim for w3lib.html
 2019-07-08 11:18:40 +05:00
Mikhail Korobov 64ff3cd6aa deprecate scrapy.utils.multipart 
It was a shim for w3lib.form, but w3lib.form is deprecated as well.
 2019-07-08 11:18:15 +05:00
Mikhail Korobov 1130711cc1 deprecate scrapy.utils.http 
ChunkedTransferMiddleware is deprecated, so decode_chunked_transfer
can be deprecated as well.
 2019-07-08 11:17:30 +05:00
Andrey Rakhmatullin 67a4000928 Work around older pyOpenSSL not having get_cipher_name or get_protocol_version_name. 2019-07-08 10:31:52 +05:00
nyov 5442c2d3c3 Updating S3FeedStorage instancing without AWS key. 2019-07-06 13:41:56 +00:00
nyov 7a398b7086 Deprecation removals for Scrapy 1.7 
Removing deprecations of 2015 and prior (pre-1.1)
 2019-07-06 13:41:56 +00:00
mikolaje 8f52dada55 PEP8 in cmdline.py 2019-07-06 19:55:09 +08:00
Daniel Graña 3d8f075b0a
Merge pull request #3840 from mabelvj/itemloader-errors 
[MRG+1] Itemloader errors #3836
 2019-07-05 08:55:43 -03:00
Mikhail Korobov 4d4bd0e823
Merge pull request #3609 from Gallaecio/2253 
Document FilesPipeline.file_path and ImagesPipeline.file_path
 2019-07-04 22:05:46 +05:00
Andrey Rakhmatullin 69b1d5d3d7 Log cipher, certificate and temp key info on establishing an SSL connection. 2019-07-04 12:13:39 +05:00
Mikhail Korobov 4c755f1ddc
Merge pull request #3692 from anubhavp28/doc-patch 
[MRG+1] doc for creating custom cache storage backend.
 2019-07-02 20:18:14 +05:00
Mikhail Korobov 9aec7856b0
Merge pull request #3794 from csalazar/whitelist-form-methods-in-fromresponse 
[MRG+1] Fix form methods in FormRequest.from_response (#3777)
 2019-07-02 20:08:14 +05:00
Claudio Salazar 2e4dc20393 Add backward compability when method=None in FormRequest 2019-06-26 21:36:28 +02:00
Daniel Graña 3adf09bc4f
Merge pull request #3563 from elacuesta/callback_kwargs 
[MRG+1] Callback kwargs
 2019-06-26 14:01:29 -03:00
Eugenio Lacuesta 312e57382c Request.cb_kwargs: update in spider middleware 2019-06-26 12:48:00 -03:00
Eugenio Lacuesta d4d68cf321 Request.cb_kwargs: update in downloader middleware 2019-06-26 12:31:41 -03:00