RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
7,790 Commits 42 Branches 132 Tags 168 MiB
Commit Graph

1187 Commits

Author SHA1 Message Date
Andrey Rakhmatullin f066257e95 Restore tests/test_proxy_connect.py and update it to modern mitmproxy. 2019-11-19 13:52:49 +05:00
Andrey Rakhmatullin e18014d84d Remove Python 2-only tests. 2019-11-19 13:52:49 +05:00
Andrey Rahmatullin 25cd7ac7ab
Merge pull request #4094 from victor-torres/invalid-url 
Improve URL schema validation on scrapy.Request initialization
 2019-11-19 13:50:08 +05:00
Andrey Rahmatullin 3408b757c1
Merge pull request #3989 from elacuesta/logformatter_error_method 
LogFormatter improvements
 2019-11-19 13:44:43 +05:00
Adrián Chaves f261cf65e9 Add missing blank lines between functions and classes 
Also fixed 2 unrelated Flake8 issues
 2019-11-18 17:16:09 +01:00
Adrián Chaves 99d8b05a0b Deprecate scrapy.utils.python.MutableChain.next 2019-11-18 10:58:47 +01:00
Marc Hernandez Cabot 0e252f5a13 fix E711 and E713 2019-11-15 19:12:43 +01:00
Mikhail Korobov 550d8046fd
Merge pull request #4157 from Gallaecio/flake8 
Add missing whitespace after ‘,’, ‘;’ or ‘:’
 2019-11-14 23:42:06 +05:00
Andrey Rakhmatullin 3b2289ad01 Rename test_non_str_url_py2 to test_bytes_url. 2019-11-14 22:53:28 +05:00
Andrey Rakhmatullin fe3a121f13 Use kwargs when calling get_func_args. 2019-11-14 22:50:53 +05:00
Adrián Chaves 058bdda0af Improve the performance of the DOWNLOAD_DELAY test 2019-11-14 16:53:20 +01:00
Andrey Rakhmatullin e291460db6 Fix flake8-detected errors. 2019-11-14 15:24:37 +05:00
Adrián Chaves 1a4a77d49f Remove Python 2 check from MutableChainTest 2019-11-14 10:24:31 +01:00
Andrey Rakhmatullin 87c23ba22d Remove Py2-only code that checks sys.version_info. 2019-11-14 13:45:39 +05:00
Andrey Rakhmatullin a138fb05d4 Replace to_native_str calls with to_unicode. 2019-11-14 13:45:39 +05:00
Andrey Rakhmatullin 92ffd2f249 Simplify some more imports. 2019-11-14 13:45:39 +05:00
Andrey Rakhmatullin 85e79ae792 Remove cStringIO imports. 2019-11-14 13:45:39 +05:00
Andrey Rakhmatullin 1d7c8cb0b1 Remove six.PY2 and six.PY3 conditionals. 2019-11-14 13:45:37 +05:00
Adrián Chaves 33ef24c757 Add missing whitespace after ‘,’, ‘;’ or ‘:’ 2019-11-13 10:52:05 +01:00
Andrey Rahmatullin c911e80209
Merge pull request #4088 from ammarnajjar/4086-constructor-initializer-docs 
docs: use __init__ method instead of constructor
 2019-11-12 21:48:16 +05:00
Andrey Rahmatullin c91390558d
Merge pull request #4074 from purvaudai/master 
Added Pathlib.Path support: Issue #3731
 2019-11-12 21:43:28 +05:00
purvaudai 4b8b0345e5 Mades Changes as per review 2019-11-12 18:17:15 +05:30
Andrey Rahmatullin 93385e647a
Merge pull request #4123 from elacuesta/utils-local-cache-limit 
Fix LocalCache limit issue, add tests
 2019-11-12 16:05:22 +05:00
purvaudai 50eaabe1fc Added Test 2019-11-11 20:00:26 +05:30
purvaudai f39ff49458 Added Test 2019-11-11 18:54:21 +05:30
purvaudai 970c3be160 Added Test 2019-11-11 18:34:15 +05:30
purvaudai 9e6e2dde2b Adding test 2019-11-11 16:10:37 +05:30
purvaudai 0042c389eb Adding test 2019-11-11 15:57:58 +05:30
purvaudai 084a1cda6d Adding test 2019-11-11 15:41:00 +05:30
Eugenio Lacuesta 07fa768d33
Merge branch 'remove-six-code' of github.com:scrapy/scrapy into remove-six-code 2019-11-08 22:17:12 -03:00
Eugenio Lacuesta b6bbb28197
PEP8 adjustments 2019-11-08 22:13:03 -03:00
Eugenio Lacuesta 342e3b5bd5
Merge remote-tracking branch 'upstream/master' into remove-six-code 2019-11-08 12:29:15 -03:00
Adrián Chaves 1df5755699 Set the bases for testing code examples from the documentation 2019-11-08 16:00:10 +01:00
elacuesta 44f19df311
[test] Update mitmproxy version 
Co-Authored-By: Adrián Chaves <adrian@chaves.io>
 2019-11-08 11:32:50 -03:00
Marc Hernández c377c14e32 Fix W391 Blank line at end of file (#4137) 2019-11-07 17:47:35 +01:00
elacuesta fe31695ba0
Remove unused import (urllib.parse.unquote) 
Co-Authored-By: Adrián Chaves <adrian@chaves.io>
 2019-11-05 15:36:19 -03:00
Eugenio Lacuesta 613c66a034
Do not override built-in max function 2019-11-05 09:45:51 -03:00
elacuesta fed9fbe62d
Update tests/test_utils_datatypes.py 
Co-Authored-By: Adrián Chaves <adrian@chaves.io>
 2019-11-04 15:34:27 -03:00
Eugenio Lacuesta 439a3e59b8
Fix scrapy.utils.datatypes.LocalCache limit issue 2019-11-04 10:35:58 -03:00
Eugenio Lacuesta 00b793dc59
Remove elluding six occurrences 2019-11-03 12:26:38 -03:00
Eugenio Lacuesta 5797aefd4c
Remove six.assertCountEqual 2019-11-03 12:18:35 -03:00
Eugenio Lacuesta 586b25d27e
Remove six types 2019-11-03 12:10:37 -03:00
Eugenio Lacuesta 7f3cb05d8e
Remove metaclass-related six code 2019-11-03 12:03:02 -03:00
Eugenio Lacuesta cca1959c07
Merge remote-tracking branch 'upstream/remove-py2-tests' into remove-six-code 2019-11-03 01:20:08 -03:00
Eugenio Lacuesta d72444b9c8
Remove more six imports 2019-11-03 01:11:28 -03:00
Eugenio Lacuesta eaeaa40b99
Remove six.PY* checks 2019-11-03 01:08:08 -03:00
Eugenio Lacuesta 5d8abdde59
Remove six.text_type from tests 2019-11-03 01:01:23 -03:00
Eugenio Lacuesta 54a786b102
Remove six imports 2019-11-03 00:58:47 -03:00
Eugenio Lacuesta 1aba513693
Remove six.iter* occurrences 2019-11-03 00:26:44 -03:00
Eugenio Lacuesta 5ab0f189ce
Remove six.moves occurrences from tests 2019-11-03 00:01:09 -03:00
Eugenio Lacuesta c0bfaef37a
Remove __future__ imports from tests 2019-11-02 23:27:04 -03:00
Andrey Rakhmatullin 350aa67c3d Rename tests/py3-ignores.txt to tests/ignores.txt. 2019-11-01 19:52:57 +05:00
Andrey Rakhmatullin 4b0cdf7f3e Use self.proc.communicate() after killing mitmdump. 2019-11-01 19:51:52 +05:00
Andrey Rakhmatullin 3c9963ab04 Only xfail test_https_connect_tunnel_error on 3.6+. 2019-11-01 19:47:12 +05:00
Andrey Rakhmatullin e0c5c72496 Improve the test_https_tunnel_without_leak_proxy_authorization_header change. 2019-11-01 19:46:19 +05:00
Andrey Rakhmatullin 5eb01b617d Use an older mitmproxy for py3.5. 2019-10-31 23:21:30 +05:00
Andrey Rakhmatullin a5eb59b92d Fix test_proxy_connect.py for py3.5. 2019-10-31 23:21:14 +05:00
Adrián Chaves f02c3d1dcf
Use communicate() instead of wait() after killing the mock server (#4095) 2019-10-31 13:31:33 +01:00
Andrey Rakhmatullin 7299e91b1f Remove Py2-only code that checks sys.version_info. 2019-10-31 15:22:53 +05:00
Andrey Rakhmatullin 397e883556 Replace to_native_str calls with to_unicode. 2019-10-31 15:22:53 +05:00
Andrey Rakhmatullin 75b1d051d9 Simplify some more imports. 2019-10-31 15:22:45 +05:00
Andrey Rakhmatullin 3ac4b430ae Remove an unused six import. 2019-10-31 15:20:28 +05:00
Andrey Rakhmatullin cea2f5e244 Remove cStringIO imports. 2019-10-31 15:07:54 +05:00
Andrey Rakhmatullin bbd9f4be90 Remove six.PY2 and six.PY3 conditionals. 2019-10-31 15:07:10 +05:00
Andrey Rakhmatullin 186f9d88ac Fix the skip message for test_download_gzip_response. 2019-10-31 15:03:44 +05:00
Andrey Rakhmatullin 439e37fc7b Mark failing proxy tests. 2019-10-31 15:03:44 +05:00
Andrey Rakhmatullin b0d6f4917d Restore tests/test_proxy_connect.py and update it to modern mitmproxy. 2019-10-31 15:03:44 +05:00
Andrey Rakhmatullin b44bd6f825 Remove Python 2-only tests. 2019-10-31 15:03:09 +05:00
Adrián Chaves 15c55d0c1d
Remove LevelDB support (#4112) 2019-10-31 10:47:29 +01:00
Andrey Rahmatullin 229e722a03 Initial Python 2 removal (#4091) 2019-10-31 10:46:02 +01:00
Benjamin Ooghe-Tabanou 6d6da78eda Add a keep_fragments parameter to the request_fingerprint function (#4104) 2019-10-30 09:13:36 +01:00
Amardeep Bhowmick 66cbceeb0a Fix redirection error when the Location header value starts with 3 slashes (#4042) 2019-10-30 09:09:12 +01:00
Andrey Rahmatullin 18b808b2e9
Merge pull request #4092 from further-reading/master 
Add Python 3.8 official support
 2019-10-29 16:30:58 +05:00
Roy 93e3dc1b82 [test_downloadermiddleware_httpcache.py] Cleaning text 
https://github.com/scrapy/scrapy/issues/4085
 2019-10-28 16:12:03 +00:00
Roy b73d217de5 [test_downloadermiddleware_httpcache.py] Fixing pytest mark behaviour 
https://github.com/scrapy/scrapy/issues/4085
 2019-10-28 12:55:54 +00:00
Roy c51fb959e2 [test_downloadermiddleware_httpcache] Fixing pytest skip behaviour 
https://github.com/scrapy/scrapy/issues/4085
 2019-10-28 12:36:54 +00:00
Roy 4432136fff [test_downloadermiddleware_httpcache] Fixing pytest skip behaviour 
https://github.com/scrapy/scrapy/issues/4085
 2019-10-28 12:22:21 +00:00
Roy 16bb3ac20d [test_downloadermiddleware_httpcache] Using skipif approach 
https://github.com/scrapy/scrapy/issues/4085
 2019-10-28 11:24:09 +00:00
elacuesta 7731814cc2 ItemLoader: improve handling of initial item (#4036) 2019-10-28 10:53:53 +01:00
Roy 20ea912513 [test_downloadermiddleware_httpcache] Making xfails more informative 
https://github.com/scrapy/scrapy/issues/4085
 2019-10-27 18:52:01 +00:00
Roy 70b2854590 [test_downloadermiddleware_httpcache] Making xfails more informative 
https://github.com/scrapy/scrapy/issues/4085
 2019-10-27 18:51:26 +00:00
Roy b3df0a8415 [test_downloadermiddleware_httpcache] Adding xfails to impacted tests following hack fix 
https://github.com/scrapy/scrapy/issues/4085
 2019-10-27 18:28:47 +00:00
Roy 11942c436c [test_downloadermiddleware_httpcache] Trying hack to handle systemerror whjen importing leveldb 
https://github.com/scrapy/scrapy/issues/4085
 2019-10-27 18:02:13 +00:00
Roy deacd34c8d [test_downloadermiddleware_httpcache] Attempting to add xfail for leveldb related tests 
https://github.com/scrapy/scrapy/issues/4085
 2019-10-27 17:39:47 +00:00
Roy Healy 4068797558
Update test_downloadermiddleware_httpcache.py 
Adding xfail denoting that leveldb is not supported in 3.8
 2019-10-27 17:02:17 +00:00
Victor Torres f701f5b0db fix #2552 by improving request schema check on its initialization 2019-10-22 10:48:02 -03:00
Ammar Najjar 7a84a4bdba
docs: use "constructor" for from_settings() & rom_crawler() factory methods 
Issue #4086
 2019-10-22 15:31:34 +02:00
Adrián Chaves 1d5c270ce8
Fix dangling file descriptor in FeedExporter when FEED_STORE_EMPTY is False (#4023) 2019-10-22 15:12:52 +02:00
purvaudai 4184bac068 Added Pathlib.Path test 2019-10-22 16:57:14 +05:30
purvaudai 85f56a92f0 Added Pathlib.Path test 2019-10-22 16:43:17 +05:30
purvaudai 7031e3a124 Added Pathlib.Path test 2019-10-22 16:31:14 +05:30
purvaudai cd4c211f4b Added Pathlib.Path test 2019-10-22 15:38:06 +05:30
purvaudai a776554282 Added Pathlib.Path test 2019-10-22 15:31:55 +05:30
purvaudai ad96d6ef59 Added Pathlib.Path test correctly 2019-10-22 14:53:59 +05:30
purvaudai 2ee38e8ddb Added Pathlib.Path test 2019-10-22 14:43:47 +05:30
Ammar Najjar da8cd9448d
docs: always surround __init__ with `` in docs 
Issue #4086
 2019-10-21 19:56:58 +02:00
Ammar Najjar 68a7d05ed8
docs: use __init__ method instead of constructor 
Issue #4086
 2019-10-21 15:42:24 +02:00
Eugenio Lacuesta e1fa1fd8ad
TextResponse.follow_all: skip invalid links 2019-10-10 00:37:30 -03:00
Eugenio Lacuesta 5f168cd459
Response.follow_all 2019-10-08 16:32:56 -03:00
Eugenio Lacuesta e0fabab5cc
Fix TypeError when using DummyStatsCollector 2019-10-01 18:03:22 -03:00
Eugenio Lacuesta 07a31b13db
Update LogFormatter tests 2019-10-01 17:55:57 -03:00
Adrián Chaves 7f4f98fd38 Provide complete API documentation coverage of scrapy.linkextractors 2019-09-30 18:22:28 +02:00
s-sanjay 2c14692e60 remove .keys() to avoid creating a tmp list/keyview obj (#4031) 
Also add --verbose and --nolinks for code coverage
 2019-09-27 09:56:43 +02:00
Mikhail Korobov 31c631f9ee
Merge pull request #3999 from Gallaecio/documentation-coverage 
Provide complete API documentation coverage of scrapy.item
 2019-09-25 16:13:13 +01:00
Adrián Chaves 1236e9e81e Provide complete API documentation coverage of scrapy.item 2019-09-25 13:24:56 +02:00
Mikhail Korobov 1364e92007
Merge pull request #4015 from elacuesta/remove_deprecated_xlib 
Remove deprecated xlib module
 2019-09-20 15:28:07 +01:00
Mikhail Korobov 17c0d72d04
Merge pull request #4016 from elacuesta/rule_default_link_extractor 
Crawling rules: make link extractors optional
 2019-09-20 15:27:17 +01:00
Eugenio Lacuesta 27436cbbc9
[test] LogFormatter.error 2019-09-16 12:12:10 -03:00
watsta 0b52fa6ca9 LogFormatter: Add the ability to skip log messages (#3987) 2019-09-16 14:12:04 +02:00
OmarFarrag 13735bcf34 Disallow media extensions unregistered with IANA (#3954) 
Co-Authored-By: s-sanjay <sanjay537@gmail.com>
 2019-09-16 14:04:06 +02:00
Eugenio Lacuesta 21ad8e20b9
Crawling rules: make link extractors optional 2019-09-13 17:32:42 -03:00
Eugenio Lacuesta c5612f387b
Remove deprecated xlib module 2019-09-13 14:32:05 -03:00
Mikhail Korobov 534de7395d
Merge pull request #3988 from elacuesta/contracts_cb_kwargs 
CallbackKeywordArgumentsContract
 2019-09-08 04:23:16 +05:00
OmarFarrag 0e8770a2f4 test for files pipeline ftp store 2019-09-06 15:47:57 +02:00
Eugenio Lacuesta b92b114633
[test] cb_kwargs contract 2019-08-31 02:44:09 -03:00
Marc Hernández ace2df3d14 Fix JSONRequest naming (#3982) 2019-08-30 11:03:44 +02:00
Mikhail Korobov ede91478e5
Merge pull request #3966 from anubhavp28/robotstxt_useragent 
Adds ROBOTSTXT_USER_AGENT setting
 2019-08-28 22:00:17 +05:00
Anubhav Patel 3a7b949d6d Adds integration with Protego robots.txt parser (#3935) 2019-08-27 09:41:31 +02:00
Anubhav Patel 00fe05e536 adds ROBOTSTXT_USER_AGENT setting 2019-08-19 09:24:16 +05:30
Andrey Rakhmatullin aaa5229e5d Fixes and improvements for DOWNLOADER_CLIENT_TLS_CIPHERS. 2019-08-13 16:56:26 +05:00
Andrey Rakhmatullin 9a8edf2bf1 Tests for setting SSL ciphers. 2019-08-13 16:53:19 +05:00
Mikhail Korobov 73d1b4b748
Merge pull request #3939 from Gallaecio/improve-scrapes-contract 
Report all missing fields when a scrapes contract fails
 2019-08-08 13:49:09 +05:00
Mikhail Korobov b4556d6508
Merge pull request #3941 from starrify/ftp-storage-uri-unquote 
[MRG+1] Added: Properly handling quoted passwords in FEED_URI for FTP
 2019-08-08 13:38:17 +05:00
Adrián Chaves 9119798a5c Add test coverage for contract failures involving multiple missing fields 2019-08-08 09:52:03 +02:00
Marc Hernández d76b6944c9 Create Request from curl command (#3862) 2019-08-08 09:43:42 +02:00
elacuesta 5dbeece8da [MRG+1] Drop py34 support - Update CI envs (#3892) 
* Drop py34 support

* Travis experiments

* More Travis experiments

* Bump Twisted version for py35+ (stretch)

* Remove Debian build

* Remove pinned lxml for Py34

* Fix merge error

* Remove unused tox env

* Add environment with pinned versions for py36

* Bump minimum Twisted version in py27; Envs with pinned versions for py27 and py35

* Add botocore as extra dep for py27 tests

* Update requirements-py2.txt

* Add botocore and Pillow as extra dependencies
 2019-08-07 12:36:52 +05:00
Pengyu Chen 7b755a41a1
Added: Properly handling quoted passwords in FEED_URI for FTP 2019-08-06 15:19:57 +01:00
Anubhav Patel 8e813953bd [MRG+1] [GSoC 2019] Interface for robots.txt parsers (#3796) 
Make the robots.txt parser configurable through the new ROBOTSTXT_PARSER setting, support the Reppy and Robotexclusionrulesparser parsers, and allow implementing custom robots.txt parsers.
 2019-08-02 09:43:29 +02:00
Renne Rocha a25e09ecdd Added constrain on lxml version based on Python version 2019-07-30 23:24:41 -03:00
Mikhail Korobov 9c514b976f
Merge pull request #3450 from wRAR/tls-logging 
Log cipher, certificate and temp key info on establishing an SSL connection
 2019-07-24 01:45:40 +05:00
Eugenio Lacuesta 7e3a602d56
Suggested changes 2019-07-15 12:37:09 -03:00
Eugenio Lacuesta d5a2a70329
Fix import 2019-07-13 23:47:41 -03:00
Eugenio Lacuesta ef9a619214
More DeprecationWarnings 2019-07-13 23:36:39 -03:00
Eugenio Lacuesta eced544d64
Prevent even more DeprecationWarnings 2019-07-13 22:16:23 -03:00
Eugenio Lacuesta 40086dabb8 Prevent more DeprecationWarnings 2019-07-13 22:14:00 -03:00
Eugenio Lacuesta 92d624c161
[Tests] Prevent more TestCase DeprecationWarnings 
* assertRegexpMatches -> assertRegex
 2019-07-13 20:55:45 -03:00
Eugenio Lacuesta b714a372e2
[Tests] Prevent more DeprecationWarnings 
* assertRegexpMatches -> assertRegex
* invalid escape sequence \[
 2019-07-13 20:23:15 -03:00
Eugenio Lacuesta 09e27d2d2e
[Tests] Monkey patch unittest.TestCase to prevent DeprecationWarning(s) 2019-07-13 19:26:59 -03:00
Andrey Rakhmatullin a96a07bc76 Add a test for DOWNLOADER_CLIENT_TLS_VERBOSE_LOGGING. 2019-07-12 18:44:45 +05:00
Luiz Francisco Rodrigues da Silva df68c4b9b1 Add a setting to use active mode in FTPFeedStorage (#3829) 
Add a setting to use active mode in FTPFeedStorage

FTP servers can be configured in active and passive mode, by default
Python and Scrapy use passive mode and there was no way to use active.

This commit adds a setting FEED_STORAGE_FTP_ACTIVE to allow a feed
to be exported to a FTP server configured in active mode.
 2019-07-10 14:25:52 +02:00
nyov 5442c2d3c3 Updating S3FeedStorage instancing without AWS key. 2019-07-06 13:41:56 +00:00
nyov 7a398b7086 Deprecation removals for Scrapy 1.7 
Removing deprecations of 2015 and prior (pre-1.1)
 2019-07-06 13:41:56 +00:00
Daniel Graña 3d8f075b0a
Merge pull request #3840 from mabelvj/itemloader-errors 
[MRG+1] Itemloader errors #3836
 2019-07-05 08:55:43 -03:00
Mikhail Korobov 9aec7856b0
Merge pull request #3794 from csalazar/whitelist-form-methods-in-fromresponse 
[MRG+1] Fix form methods in FormRequest.from_response (#3777)
 2019-07-02 20:08:14 +05:00
Eugenio Lacuesta 312e57382c Request.cb_kwargs: update in spider middleware 2019-06-26 12:48:00 -03:00
Eugenio Lacuesta d4d68cf321 Request.cb_kwargs: update in downloader middleware 2019-06-26 12:31:41 -03:00
Eugenio Lacuesta 1f9f41b850 Move request.cb_kwargs tests to their own test file 2019-06-26 12:31:12 -03:00
Eugenio Lacuesta 428309ba1a Merge remote-tracking branch 'origin/master' into callback_kwargs 2019-06-26 11:03:31 -03:00
Mikhail Korobov c81d120bde
Merge pull request #3768 from Gallaecio/meta-ignore-tags 
Implement the METAREFRESH_IGNORE_TAGS setting
 2019-06-25 23:25:58 +03:00
Mabel Villalba b1e348b281 [itemloader-errors] updated Compose and MapCompose messages and added tests 
#3836
 2019-06-25 17:11:14 +02:00