RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
7,254 Commits 49 Branches 132 Tags 133 MiB
Commit Graph

7254 Commits

Author SHA1 Message Date
Mikhail Korobov cdf7889ada
Merge pull request #3923 from scrapy/pin-build-environment 
Pin Travis-ci build environment to previous default: Trusty
 2019-07-31 20:07:44 +05:00
Renne Rocha a25e09ecdd Added constrain on lxml version based on Python version 2019-07-30 23:24:41 -03:00
Daniel Graña 7333fc02aa Pin Travis-ci build environment to previous default: Trusty 
Travis-ci changed the default build environment to Xenial as explained in https://blog.travis-ci.com/2019-04-15-xenial-default-build-environment
This causes builds meant for Debian Jessie to break as noted by @wRAR in https://github.com/scrapy/scrapy/issues/3917#issuecomment-516426389

This change pins the environment to known working ubuntu trusty distribution prior to dropping Jessie support and upgrade to Xenial as base.

Closes #1369
 2019-07-30 23:16:11 -03:00
Daniel Graña b01d012b5a
Merge pull request #3920 from wRAR/fix-tls-logging 
Fix memory handling and error handling in utils.ssl.get_temp_key_info.
 2019-07-30 22:15:26 -03:00
Andrey Rakhmatullin f21dc24a26 Fix memory handling and error handling in utils.ssl.get_temp_key_info. 2019-07-30 18:16:12 +05:00
Adrián Chaves 06c093f43f
Merge pull request #3905 from lucywang000/0.001 
s3 file store persist_file should accept all supported headers
 2019-07-29 19:02:14 +02:00
Adrián Chaves 04bca6af7c
Merge pull request #3894 from KristobalJunta/fix_retry_docs 
fix default RETRY_HTTP_CODES value in docs
 2019-07-29 18:20:55 +02:00
Lucy Wang 7551689c75 s3 file store should accept all supported headers 2019-07-26 09:07:29 +08:00
Mikhail Korobov 9c514b976f
Merge pull request #3450 from wRAR/tls-logging 
Log cipher, certificate and temp key info on establishing an SSL connection
 2019-07-24 01:45:40 +05:00
Mikhail Korobov c679aefe85
Merge pull request #3660 from anubhavp28/logFormatter-doc-patch 
[MRG+1] docs for scrapy.logformatter
 2019-07-24 01:40:56 +05:00
Adrián Chaves 7843101f9a Cover Scrapy 1.7.2 in the release notes 2019-07-23 13:35:27 +02:00
Adrián Chaves bc8672c366
Merge pull request #3896 from elacuesta/fix_configparser_import 
Fix ConfigParser import in py2
 2019-07-23 11:59:53 +02:00
Eugenio Lacuesta 7e622af4e5
Fix ConfigParser import in py2 2019-07-22 14:53:17 -03:00
Kristobal Junta 43d5b5a524 fix default RETRY_HTTP_CODES value in docs 2019-07-22 10:24:14 +03:00
Adrián Chaves b8a43011e7 Cover Scrapy 1.7.1 in the release notes 2019-07-19 18:17:24 +02:00
Andrey Rakhmatullin c6453800cd Remove an unneeded if. 2019-07-18 22:17:39 +05:00
Andrey Rakhmatullin 95dd2df7b5 Drop an unused import. 2019-07-18 20:51:26 +05:00
Andrey Rakhmatullin 42743fd9dd Move tls_verbose_logging extraction from __init__ to from_settings. 2019-07-18 20:49:25 +05:00
Adrián Chaves ae4eab9843 Cover the 1.7.1 PyPI repackaging in the release notes 2019-07-18 17:02:48 +02:00
Adrián Chaves 4e23d70dd3 Bump version: 1.6.0 → 1.7.0 2019-07-18 15:38:25 +02:00
Adrián Chaves a94b5bef3a Write the 1.7 release notes and cover dropping Python 2 support in the upcoming 2.0 2019-07-18 15:37:17 +02:00
Mikhail Korobov 44eb21aa51
Merge pull request #3882 from MagdalenaDeschner/master 
add instructions about how to define output file when running scrapy …
 2019-07-18 18:12:57 +05:00
Deschner, Magdalena c44d49b238 minor PEP8 style changes 2019-07-17 13:13:52 +02:00
Maram Sumanth 0d51f9cc27 [MRG+1] Wrong value of log_count/INFO in stats (#3643) 
* Update statscollectors.py

* Update statscollectors.py

* Update statscollectors.py

* Update crawler.py

* Update crawler.py

* corrected tests

* Update test_utils_log.py

* Update crawler.py

* Update crawler.py

* interchanged order

* correced
 2019-07-17 14:36:49 +05:00
Mikhail Korobov b2c013feca
Merge pull request #3878 from elacuesta/mergedict_to_chainmap 
Deprecate scrapy.utils.datatypes.MergeDict in favor of collections.ChainMap
 2019-07-17 14:31:50 +05:00
Deschner, Magdalena 6660020ebb remove detailed description about individual settings 2019-07-17 11:30:02 +02:00
Eugenio Lacuesta d7074d86d2
Change condition to raise deprecation warning 2019-07-16 14:13:45 -03:00
Mikhail Korobov 377d8a7be7
Merge pull request #3877 from elacuesta/tests_deprecation_warnings 
Prevent deprecation warnings
 2019-07-16 21:58:44 +05:00
Deschner, Magdalena e892a484e8 add instructions about how to define output file when running scrapy from script instead of cmd 2019-07-16 13:53:56 +02:00
Eugenio Lacuesta 7e3a602d56
Suggested changes 2019-07-15 12:37:09 -03:00
Eugenio Lacuesta d5a2a70329
Fix import 2019-07-13 23:47:41 -03:00
Eugenio Lacuesta ef9a619214
More DeprecationWarnings 2019-07-13 23:36:39 -03:00
Eugenio Lacuesta 62f3e22481
Deprecate scrapy.utils.datatypes.MergeDict in favor of collections.ChainMap 2019-07-13 22:51:38 -03:00
Eugenio Lacuesta eced544d64
Prevent even more DeprecationWarnings 2019-07-13 22:16:23 -03:00
Eugenio Lacuesta 40086dabb8 Prevent more DeprecationWarnings 2019-07-13 22:14:00 -03:00
Eugenio Lacuesta 92d624c161
[Tests] Prevent more TestCase DeprecationWarnings 
* assertRegexpMatches -> assertRegex
 2019-07-13 20:55:45 -03:00
Eugenio Lacuesta c24b80e1e6
Prevent DeprecationWarning about logging.warn 2019-07-13 20:34:31 -03:00
Eugenio Lacuesta b714a372e2
[Tests] Prevent more DeprecationWarnings 
* assertRegexpMatches -> assertRegex
* invalid escape sequence \[
 2019-07-13 20:23:15 -03:00
Eugenio Lacuesta 27e63e6890
Monkey patch configparser module to ease the dropping of py2 support 2019-07-13 20:10:06 -03:00
Eugenio Lacuesta 09e27d2d2e
[Tests] Monkey patch unittest.TestCase to prevent DeprecationWarning(s) 2019-07-13 19:26:59 -03:00
Andrey Rakhmatullin a96a07bc76 Add a test for DOWNLOADER_CLIENT_TLS_VERBOSE_LOGGING. 2019-07-12 18:44:45 +05:00
Adrián Chaves fa6a0d799b
Merge pull request #3684 from Gallaecio/remove-ubuntu-topic 
Remove docs/topics/ubuntu.rst
 2019-07-12 14:03:11 +02:00
Adrián Chaves c4bcfb1aee
Merge pull request #3866 from Gallaecio/broad-crawl-memory-docs 
Extend the topic about broad crawls with information about memory usage
 2019-07-11 11:15:41 +02:00
Adrián Chaves ed487b7db3 broad-crawls.rst: Refactor the memory usage section 2019-07-11 11:14:22 +02:00
Andrey Rakhmatullin 98689b27a8 Improve the DOWNLOADER_CLIENTCONTEXTFACTORY doc. 2019-07-11 14:02:35 +05:00
Andrey Rakhmatullin 0de6ffc8e1 Fix super() call. 2019-07-11 13:12:56 +05:00
Robin 5cdf277017 Update broad-crawls.rst 
Added section on how to treat memory consumption problems of broad crawls.
 2019-07-11 08:49:24 +02:00
Luiz Francisco Rodrigues da Silva df68c4b9b1 Add a setting to use active mode in FTPFeedStorage (#3829) 
Add a setting to use active mode in FTPFeedStorage

FTP servers can be configured in active and passive mode, by default
Python and Scrapy use passive mode and there was no way to use active.

This commit adds a setting FEED_STORAGE_FTP_ACTIVE to allow a feed
to be exported to a FTP server configured in active mode.
 2019-07-10 14:25:52 +02:00
Mikhail Korobov 050a62a955
Merge pull request #3861 from Gallaecio/documentation-coverage 
Increase the documentation coverage of exceptions
 2019-07-10 10:18:34 +05:00
Adrián Chaves 68d898d493
Merge pull request #3712 from elacuesta/docs_crawlspider_link_text 
[Docs] CrawlSpider: update Rule docs
 2019-07-10 00:18:51 +02:00