RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
7,345 Commits 49 Branches 132 Tags 141 MiB
Commit Graph

985 Commits

Author SHA1 Message Date
purvaudai 2ee38e8ddb Added Pathlib.Path test 2019-10-22 14:43:47 +05:30
Eugenio Lacuesta e0fabab5cc
Fix TypeError when using DummyStatsCollector 2019-10-01 18:03:22 -03:00
s-sanjay 2c14692e60 remove .keys() to avoid creating a tmp list/keyview obj (#4031) 
Also add --verbose and --nolinks for code coverage
 2019-09-27 09:56:43 +02:00
Mikhail Korobov 31c631f9ee
Merge pull request #3999 from Gallaecio/documentation-coverage 
Provide complete API documentation coverage of scrapy.item
 2019-09-25 16:13:13 +01:00
Adrián Chaves 1236e9e81e Provide complete API documentation coverage of scrapy.item 2019-09-25 13:24:56 +02:00
Mikhail Korobov 1364e92007
Merge pull request #4015 from elacuesta/remove_deprecated_xlib 
Remove deprecated xlib module
 2019-09-20 15:28:07 +01:00
Mikhail Korobov 17c0d72d04
Merge pull request #4016 from elacuesta/rule_default_link_extractor 
Crawling rules: make link extractors optional
 2019-09-20 15:27:17 +01:00
watsta 0b52fa6ca9 LogFormatter: Add the ability to skip log messages (#3987) 2019-09-16 14:12:04 +02:00
OmarFarrag 13735bcf34 Disallow media extensions unregistered with IANA (#3954) 
Co-Authored-By: s-sanjay <sanjay537@gmail.com>
 2019-09-16 14:04:06 +02:00
Eugenio Lacuesta 21ad8e20b9
Crawling rules: make link extractors optional 2019-09-13 17:32:42 -03:00
Eugenio Lacuesta c5612f387b
Remove deprecated xlib module 2019-09-13 14:32:05 -03:00
Mikhail Korobov 534de7395d
Merge pull request #3988 from elacuesta/contracts_cb_kwargs 
CallbackKeywordArgumentsContract
 2019-09-08 04:23:16 +05:00
Eugenio Lacuesta b92b114633
[test] cb_kwargs contract 2019-08-31 02:44:09 -03:00
Marc Hernández ace2df3d14 Fix JSONRequest naming (#3982) 2019-08-30 11:03:44 +02:00
Mikhail Korobov ede91478e5
Merge pull request #3966 from anubhavp28/robotstxt_useragent 
Adds ROBOTSTXT_USER_AGENT setting
 2019-08-28 22:00:17 +05:00
Anubhav Patel 3a7b949d6d Adds integration with Protego robots.txt parser (#3935) 2019-08-27 09:41:31 +02:00
Anubhav Patel 00fe05e536 adds ROBOTSTXT_USER_AGENT setting 2019-08-19 09:24:16 +05:30
Andrey Rakhmatullin aaa5229e5d Fixes and improvements for DOWNLOADER_CLIENT_TLS_CIPHERS. 2019-08-13 16:56:26 +05:00
Andrey Rakhmatullin 9a8edf2bf1 Tests for setting SSL ciphers. 2019-08-13 16:53:19 +05:00
Mikhail Korobov 73d1b4b748
Merge pull request #3939 from Gallaecio/improve-scrapes-contract 
Report all missing fields when a scrapes contract fails
 2019-08-08 13:49:09 +05:00
Mikhail Korobov b4556d6508
Merge pull request #3941 from starrify/ftp-storage-uri-unquote 
[MRG+1] Added: Properly handling quoted passwords in FEED_URI for FTP
 2019-08-08 13:38:17 +05:00
Adrián Chaves 9119798a5c Add test coverage for contract failures involving multiple missing fields 2019-08-08 09:52:03 +02:00
Marc Hernández d76b6944c9 Create Request from curl command (#3862) 2019-08-08 09:43:42 +02:00
elacuesta 5dbeece8da [MRG+1] Drop py34 support - Update CI envs (#3892) 
* Drop py34 support

* Travis experiments

* More Travis experiments

* Bump Twisted version for py35+ (stretch)

* Remove Debian build

* Remove pinned lxml for Py34

* Fix merge error

* Remove unused tox env

* Add environment with pinned versions for py36

* Bump minimum Twisted version in py27; Envs with pinned versions for py27 and py35

* Add botocore as extra dep for py27 tests

* Update requirements-py2.txt

* Add botocore and Pillow as extra dependencies
 2019-08-07 12:36:52 +05:00
Pengyu Chen 7b755a41a1
Added: Properly handling quoted passwords in FEED_URI for FTP 2019-08-06 15:19:57 +01:00
Anubhav Patel 8e813953bd [MRG+1] [GSoC 2019] Interface for robots.txt parsers (#3796) 
Make the robots.txt parser configurable through the new ROBOTSTXT_PARSER setting, support the Reppy and Robotexclusionrulesparser parsers, and allow implementing custom robots.txt parsers.
 2019-08-02 09:43:29 +02:00
Renne Rocha a25e09ecdd Added constrain on lxml version based on Python version 2019-07-30 23:24:41 -03:00
Mikhail Korobov 9c514b976f
Merge pull request #3450 from wRAR/tls-logging 
Log cipher, certificate and temp key info on establishing an SSL connection
 2019-07-24 01:45:40 +05:00
Eugenio Lacuesta 7e3a602d56
Suggested changes 2019-07-15 12:37:09 -03:00
Eugenio Lacuesta d5a2a70329
Fix import 2019-07-13 23:47:41 -03:00
Eugenio Lacuesta ef9a619214
More DeprecationWarnings 2019-07-13 23:36:39 -03:00
Eugenio Lacuesta eced544d64
Prevent even more DeprecationWarnings 2019-07-13 22:16:23 -03:00
Eugenio Lacuesta 40086dabb8 Prevent more DeprecationWarnings 2019-07-13 22:14:00 -03:00
Eugenio Lacuesta 92d624c161
[Tests] Prevent more TestCase DeprecationWarnings 
* assertRegexpMatches -> assertRegex
 2019-07-13 20:55:45 -03:00
Eugenio Lacuesta b714a372e2
[Tests] Prevent more DeprecationWarnings 
* assertRegexpMatches -> assertRegex
* invalid escape sequence \[
 2019-07-13 20:23:15 -03:00
Eugenio Lacuesta 09e27d2d2e
[Tests] Monkey patch unittest.TestCase to prevent DeprecationWarning(s) 2019-07-13 19:26:59 -03:00
Andrey Rakhmatullin a96a07bc76 Add a test for DOWNLOADER_CLIENT_TLS_VERBOSE_LOGGING. 2019-07-12 18:44:45 +05:00
Luiz Francisco Rodrigues da Silva df68c4b9b1 Add a setting to use active mode in FTPFeedStorage (#3829) 
Add a setting to use active mode in FTPFeedStorage

FTP servers can be configured in active and passive mode, by default
Python and Scrapy use passive mode and there was no way to use active.

This commit adds a setting FEED_STORAGE_FTP_ACTIVE to allow a feed
to be exported to a FTP server configured in active mode.
 2019-07-10 14:25:52 +02:00
nyov 5442c2d3c3 Updating S3FeedStorage instancing without AWS key. 2019-07-06 13:41:56 +00:00
nyov 7a398b7086 Deprecation removals for Scrapy 1.7 
Removing deprecations of 2015 and prior (pre-1.1)
 2019-07-06 13:41:56 +00:00
Daniel Graña 3d8f075b0a
Merge pull request #3840 from mabelvj/itemloader-errors 
[MRG+1] Itemloader errors #3836
 2019-07-05 08:55:43 -03:00
Mikhail Korobov 9aec7856b0
Merge pull request #3794 from csalazar/whitelist-form-methods-in-fromresponse 
[MRG+1] Fix form methods in FormRequest.from_response (#3777)
 2019-07-02 20:08:14 +05:00
Eugenio Lacuesta 312e57382c Request.cb_kwargs: update in spider middleware 2019-06-26 12:48:00 -03:00
Eugenio Lacuesta d4d68cf321 Request.cb_kwargs: update in downloader middleware 2019-06-26 12:31:41 -03:00
Eugenio Lacuesta 1f9f41b850 Move request.cb_kwargs tests to their own test file 2019-06-26 12:31:12 -03:00
Eugenio Lacuesta 428309ba1a Merge remote-tracking branch 'origin/master' into callback_kwargs 2019-06-26 11:03:31 -03:00
Mikhail Korobov c81d120bde
Merge pull request #3768 from Gallaecio/meta-ignore-tags 
Implement the METAREFRESH_IGNORE_TAGS setting
 2019-06-25 23:25:58 +03:00
Mabel Villalba b1e348b281 [itemloader-errors] updated Compose and MapCompose messages and added tests 
#3836
 2019-06-25 17:11:14 +02:00
Mabel Villalba a753ea7e16 [itemloader-errors] added test for Compose and MapCompose 
#3836
 2019-06-25 14:35:49 +02:00
Mabel Villalba 99eb03a810 [itemloader-errors] adapted compose test for the new error being returned by compose 
#3836
 2019-06-25 14:24:27 +02:00