RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
7,267 Commits 50 Branches 132 Tags 142 MiB
Commit Graph

7267 Commits

Author SHA1 Message Date
Mikhail Korobov 73d1b4b748
Merge pull request #3939 from Gallaecio/improve-scrapes-contract 
Report all missing fields when a scrapes contract fails
 2019-08-08 13:49:09 +05:00
Mikhail Korobov b4556d6508
Merge pull request #3941 from starrify/ftp-storage-uri-unquote 
[MRG+1] Added: Properly handling quoted passwords in FEED_URI for FTP
 2019-08-08 13:38:17 +05:00
Adrián Chaves 9119798a5c Add test coverage for contract failures involving multiple missing fields 2019-08-08 09:52:03 +02:00
Marc Hernández d76b6944c9 Create Request from curl command (#3862) 2019-08-08 09:43:42 +02:00
elacuesta 5dbeece8da [MRG+1] Drop py34 support - Update CI envs (#3892) 
* Drop py34 support

* Travis experiments

* More Travis experiments

* Bump Twisted version for py35+ (stretch)

* Remove Debian build

* Remove pinned lxml for Py34

* Fix merge error

* Remove unused tox env

* Add environment with pinned versions for py36

* Bump minimum Twisted version in py27; Envs with pinned versions for py27 and py35

* Add botocore as extra dep for py27 tests

* Update requirements-py2.txt

* Add botocore and Pillow as extra dependencies
 2019-08-07 12:36:52 +05:00
Pengyu Chen 7b755a41a1
Added: Properly handling quoted passwords in FEED_URI for FTP 2019-08-06 15:19:57 +01:00
Adrián Chaves bff335cf7f Improve the error message in contract failures due to multiple missing fields 2019-08-05 15:47:58 +02:00
tpeng a8621bbc29 show all the missing field when scrapes contract fails 2019-08-05 15:27:31 +02:00
Daniel Graña eef1732374
Merge pull request #3934 from anubhavp28/typo-fix 
[MRG+1] Fixes typo
 2019-08-05 09:39:21 -03:00
Anubhav Patel 9a4cd94244 fixes typo 2019-08-03 22:46:06 +05:30
Anubhav Patel 8e813953bd [MRG+1] [GSoC 2019] Interface for robots.txt parsers (#3796) 
Make the robots.txt parser configurable through the new ROBOTSTXT_PARSER setting, support the Reppy and Robotexclusionrulesparser parsers, and allow implementing custom robots.txt parsers.
 2019-08-02 09:43:29 +02:00
Adrián Chaves a12e8251e0 Cover Scrapy 1.7.3 in the release notes 2019-08-01 17:10:31 +02:00
sbs2001 783d61d32a [MRG+1] Update _monkeypatches.py (#3907) 
* Update _monkeypatches.py

The workarounds are not required assuming the bugs regarding urlparse are absent in  Python versions >2.7. We already exit the program if Python  version<2.7 in the __init__.py(line 17).The monkeypatches are deployed after this check at line 27  in  the __init__.py .

* Update _monkeypatches.py

Added the second workaround.

* Update _monkeypatches.py

* Update _monkeypatches.py
 2019-08-01 13:41:27 +05:00
Mikhail Korobov cdf7889ada
Merge pull request #3923 from scrapy/pin-build-environment 
Pin Travis-ci build environment to previous default: Trusty
 2019-07-31 20:07:44 +05:00
Renne Rocha a25e09ecdd Added constrain on lxml version based on Python version 2019-07-30 23:24:41 -03:00
Daniel Graña 7333fc02aa Pin Travis-ci build environment to previous default: Trusty 
Travis-ci changed the default build environment to Xenial as explained in https://blog.travis-ci.com/2019-04-15-xenial-default-build-environment
This causes builds meant for Debian Jessie to break as noted by @wRAR in https://github.com/scrapy/scrapy/issues/3917#issuecomment-516426389

This change pins the environment to known working ubuntu trusty distribution prior to dropping Jessie support and upgrade to Xenial as base.

Closes #1369
 2019-07-30 23:16:11 -03:00
Daniel Graña b01d012b5a
Merge pull request #3920 from wRAR/fix-tls-logging 
Fix memory handling and error handling in utils.ssl.get_temp_key_info.
 2019-07-30 22:15:26 -03:00
Andrey Rakhmatullin f21dc24a26 Fix memory handling and error handling in utils.ssl.get_temp_key_info. 2019-07-30 18:16:12 +05:00
Adrián Chaves 06c093f43f
Merge pull request #3905 from lucywang000/0.001 
s3 file store persist_file should accept all supported headers
 2019-07-29 19:02:14 +02:00
Adrián Chaves 04bca6af7c
Merge pull request #3894 from KristobalJunta/fix_retry_docs 
fix default RETRY_HTTP_CODES value in docs
 2019-07-29 18:20:55 +02:00
Lucy Wang 7551689c75 s3 file store should accept all supported headers 2019-07-26 09:07:29 +08:00
Mikhail Korobov 9c514b976f
Merge pull request #3450 from wRAR/tls-logging 
Log cipher, certificate and temp key info on establishing an SSL connection
 2019-07-24 01:45:40 +05:00
Mikhail Korobov c679aefe85
Merge pull request #3660 from anubhavp28/logFormatter-doc-patch 
[MRG+1] docs for scrapy.logformatter
 2019-07-24 01:40:56 +05:00
Adrián Chaves 7843101f9a Cover Scrapy 1.7.2 in the release notes 2019-07-23 13:35:27 +02:00
Adrián Chaves bc8672c366
Merge pull request #3896 from elacuesta/fix_configparser_import 
Fix ConfigParser import in py2
 2019-07-23 11:59:53 +02:00
Eugenio Lacuesta 7e622af4e5
Fix ConfigParser import in py2 2019-07-22 14:53:17 -03:00
Kristobal Junta 43d5b5a524 fix default RETRY_HTTP_CODES value in docs 2019-07-22 10:24:14 +03:00
Adrián Chaves b8a43011e7 Cover Scrapy 1.7.1 in the release notes 2019-07-19 18:17:24 +02:00
Andrey Rakhmatullin c6453800cd Remove an unneeded if. 2019-07-18 22:17:39 +05:00
Andrey Rakhmatullin 95dd2df7b5 Drop an unused import. 2019-07-18 20:51:26 +05:00
Andrey Rakhmatullin 42743fd9dd Move tls_verbose_logging extraction from __init__ to from_settings. 2019-07-18 20:49:25 +05:00
Adrián Chaves ae4eab9843 Cover the 1.7.1 PyPI repackaging in the release notes 2019-07-18 17:02:48 +02:00
Adrián Chaves 4e23d70dd3 Bump version: 1.6.0 → 1.7.0 2019-07-18 15:38:25 +02:00
Adrián Chaves a94b5bef3a Write the 1.7 release notes and cover dropping Python 2 support in the upcoming 2.0 2019-07-18 15:37:17 +02:00
Mikhail Korobov 44eb21aa51
Merge pull request #3882 from MagdalenaDeschner/master 
add instructions about how to define output file when running scrapy …
 2019-07-18 18:12:57 +05:00
Deschner, Magdalena c44d49b238 minor PEP8 style changes 2019-07-17 13:13:52 +02:00
Maram Sumanth 0d51f9cc27 [MRG+1] Wrong value of log_count/INFO in stats (#3643) 
* Update statscollectors.py

* Update statscollectors.py

* Update statscollectors.py

* Update crawler.py

* Update crawler.py

* corrected tests

* Update test_utils_log.py

* Update crawler.py

* Update crawler.py

* interchanged order

* correced
 2019-07-17 14:36:49 +05:00
Mikhail Korobov b2c013feca
Merge pull request #3878 from elacuesta/mergedict_to_chainmap 
Deprecate scrapy.utils.datatypes.MergeDict in favor of collections.ChainMap
 2019-07-17 14:31:50 +05:00
Deschner, Magdalena 6660020ebb remove detailed description about individual settings 2019-07-17 11:30:02 +02:00
Eugenio Lacuesta d7074d86d2
Change condition to raise deprecation warning 2019-07-16 14:13:45 -03:00
Mikhail Korobov 377d8a7be7
Merge pull request #3877 from elacuesta/tests_deprecation_warnings 
Prevent deprecation warnings
 2019-07-16 21:58:44 +05:00
Deschner, Magdalena e892a484e8 add instructions about how to define output file when running scrapy from script instead of cmd 2019-07-16 13:53:56 +02:00
Eugenio Lacuesta 7e3a602d56
Suggested changes 2019-07-15 12:37:09 -03:00
Eugenio Lacuesta d5a2a70329
Fix import 2019-07-13 23:47:41 -03:00
Eugenio Lacuesta ef9a619214
More DeprecationWarnings 2019-07-13 23:36:39 -03:00
Eugenio Lacuesta 62f3e22481
Deprecate scrapy.utils.datatypes.MergeDict in favor of collections.ChainMap 2019-07-13 22:51:38 -03:00
Eugenio Lacuesta eced544d64
Prevent even more DeprecationWarnings 2019-07-13 22:16:23 -03:00
Eugenio Lacuesta 40086dabb8 Prevent more DeprecationWarnings 2019-07-13 22:14:00 -03:00
Eugenio Lacuesta 92d624c161
[Tests] Prevent more TestCase DeprecationWarnings 
* assertRegexpMatches -> assertRegex
 2019-07-13 20:55:45 -03:00
Eugenio Lacuesta c24b80e1e6
Prevent DeprecationWarning about logging.warn 2019-07-13 20:34:31 -03:00