RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
7,653 Commits 42 Branches 132 Tags 168 MiB
Commit Graph

4573 Commits

Author SHA1 Message Date
Eugenio Lacuesta 931b7e68d3
Update FileDownloadHandler test 2019-12-23 09:50:28 -03:00
Eugenio Lacuesta 310cc081d0
Merge branch 'master' into from_crawler_downloader_handlers 2019-12-21 17:24:45 -03:00
Mikhail Korobov 5577d4d2be
Merge pull request #3869 from elacuesta/detect_return_in_generator_callbacks 
[MRG+1][Py3] Check for 'return' with arguments inside generators
 2019-12-20 22:46:46 +05:00
Mikhail Korobov fb3fb17c28
Merge pull request #4045 from Gallaecio/documentation-coverage 
Deprecate FilteringLinkExtractor direct usage; add API docs for scrapy.linkextractors
 2019-12-20 02:01:45 +05:00
Adrián Chaves e22c0c27d9 Revert "Improve FilteringLinkExtractor.__new__" 
This reverts commit ee9881d270.
 2019-12-19 12:15:54 +01:00
Marc Hernandez Cabot a5de2c64e6 fix W291, W292, W293 (whitespaces) 2019-12-18 16:33:46 +01:00
elacuesta 916382e109 Add errback parameter to scrapy.spiders.crawl.Rule (#4000) 
* Add errback parameter to scrapy.spiders.crawl.Rule

* CrawlSpider: optimize by reducing iterations

* [test] Rule.errback

* [doc] Rule.errback

* [doc] Use autoclass in docs/topics/spiders.rst

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Rule.process_links takes a list

* Fix aesthetic issue reported by Flake8
 2019-12-18 20:05:33 +05:00
Mikhail Korobov e90f276903
Merge pull request #4198 from wRAR/deprecate-noconnect 
Deprecate the HTTPS proxy noconnect mode.
 2019-12-18 16:11:24 +05:00
Adrián Chaves 1f689b50c6 Merge remote-tracking branch 'origin/documentation-coverage' into documentation-coverage 2019-12-18 12:09:34 +01:00
Adrián Chaves ee9881d270 Improve FilteringLinkExtractor.__new__ 2019-12-18 12:08:34 +01:00
Adrián Chaves 607815dfc2
Merge branch 'master' into documentation-coverage 2019-12-17 16:54:27 +01:00
Andrey Rahmatullin 4be2bbfe75
Merge pull request #4239 from Apuyuseng/master 
Fix mail attachs tcmime *** (#4229)
 2019-12-17 19:43:30 +05:00
Marc Hernández 63cf5c75c8 Fix E502: backslash is redundant between brackets (#4238) 2019-12-17 13:53:15 +01:00
apu 7d0096da6e
Fix mail attachs tcmime *** (#4229) 
When the file name consists of alphanumeric characters, it is normal to receive the attachment name.
However，However, problems will occur if the file name is changed to Chinese.
This has nothing to do with the file type
 2019-12-17 09:47:01 +08:00
Eugenio Lacuesta 451e7a616e
Scan callbacks/errbacks for return statements with values different than None 2019-12-16 11:30:40 -03:00
Adrián Chaves b5c4c2cae8
Keep 2 spaces between code and inline comments (#4195) 2019-12-13 14:20:48 +01:00
Adrián Chaves 1fc2b140c1
Merge branch 'master' into documentation-coverage 2019-12-05 14:43:36 +01:00
Andrey Rahmatullin 076f0764b7
Merge pull request #4121 from scrapy/remove-six-code 
Remove six-related code and __future__ imports
 2019-12-05 18:20:14 +05:00
Mikhail Korobov 250da28952
Merge pull request #4170 from mabelvj/4133-handle-start_url 
Raise error when start_url found instead of start_urls.
 2019-12-05 17:47:03 +05:00
Grammy Jiang 9b4b43f8ac
Convert the relative imports to absolute imports 
This commits converts the relative imports to absolute imports in the
entire package
 2019-12-05 11:25:19 +11:00
Eugenio Lacuesta 2c010152c3
Merge remote-tracking branch 'upstream/master' into remove-six-code 2019-12-04 15:43:02 -03:00
Grammy Jiang 74627033c4 Remove the used import and re-arrange the imports (#4208) 
This commit removes unused import and re-arrange the imports in cookies
module
 2019-12-04 14:24:14 +01:00
Grammy Jiang 702333478d Re-arrange the imports in httpcache module (#4209) 
This commit re-arrange the imports in httpcache module to follow pep8
 2019-12-04 14:23:28 +01:00
Grammy Jiang 5d8d4bb7d7 Re-arrange the imports in the httpproxy module (#4210) 
This commit re-arranges the imports in the httpproxy module to follow
pep8
 2019-12-04 14:22:10 +01:00
Eugenio Lacuesta e43f37fff3
Pass args/kwargs in S3DownloadHandler.from_crawler, update tests 2019-12-02 17:36:31 -03:00
Eugenio Lacuesta 3d77f74e40
Download handlers: from_crawler factory method, take crawler instead of settings in __init__ 2019-12-02 17:36:31 -03:00
Grammy Jiang d1cdfb4701 Use pprint.pformat on overridden settings (#4199) 
Keeps consistency with scrapy.middleware
 2019-11-29 09:13:57 +01:00
Andrey Rakhmatullin 63546cbf3e Deprecate the HTTPS proxy noconnect mode. 2019-11-27 22:42:52 +05:00
Adrián Chaves 6d9ed6146d
Merge branch 'master' into remove-six-code 2019-11-25 10:34:21 +01:00
Eugenio Lacuesta 40b5cfc0a4
Item loaders: allow single-argument processors (unbound methods) 2019-11-22 20:47:22 -03:00
Eugenio Lacuesta 5bab3c0261
Merge remote-tracking branch 'upstream/master' into remove-six-code 2019-11-22 12:12:29 -03:00
Mabel Villalba 070b3a4e84
Merge branch 'master' into 4133-handle-start_url 2019-11-21 17:10:31 +01:00
Mabel Villalba 1718e450ef [start_url] Fixes #4133: Raise AttributeError error when empty 'start_urls' and 'start_url' found. Added test. 2019-11-21 17:06:47 +01:00
Adrián Chaves b232881356 Restore intentional import of unused objects 2019-11-21 14:30:10 +01:00
Adrián Chaves a2bf340bab Remove unused imports 2019-11-21 14:21:17 +01:00
Eugenio Lacuesta 714e0d3ac9
Merge remote-tracking branch 'upstream/master' into remove-six-code 2019-11-20 12:41:38 -03:00
Mikhail Korobov 65e2fb7d89
Merge pull request #4153 from Gallaecio/lgtm 
MutableChain: return self from __iter__
 2019-11-20 19:46:20 +05:00
Mikhail Korobov d69e788fec
Merge pull request #4172 from Gallaecio/intersphinx 
Use InterSphinx to link to the Twisted documentation
 2019-11-20 19:38:50 +05:00
Mikhail Korobov e96ca89a1d
Merge pull request #4174 from Gallaecio/flake8 
Add missing blank lines between functions and classes
 2019-11-20 19:34:04 +05:00
Eugenio Lacuesta 05785c1c17
Merge remote-tracking branch 'upstream/master' into remove-six-code 2019-11-19 11:01:34 -03:00
Andrey Rahmatullin 25cd7ac7ab
Merge pull request #4094 from victor-torres/invalid-url 
Improve URL schema validation on scrapy.Request initialization
 2019-11-19 13:50:08 +05:00
Andrey Rahmatullin 3408b757c1
Merge pull request #3989 from elacuesta/logformatter_error_method 
LogFormatter improvements
 2019-11-19 13:44:43 +05:00
Andrey Rahmatullin e829c47eb8
Merge pull request #3937 from sbs2001/patch-10 
Update reactor.py, updated 'if' sequencing .
 2019-11-19 11:20:56 +05:00
Adrián Chaves f261cf65e9 Add missing blank lines between functions and classes 
Also fixed 2 unrelated Flake8 issues
 2019-11-18 17:16:09 +01:00
Adrián Chaves e84cb18ca0 Use InterSphinx to link to the Twisted documentation 2019-11-18 15:50:45 +01:00
Adrián Chaves 99d8b05a0b Deprecate scrapy.utils.python.MutableChain.next 2019-11-18 10:58:47 +01:00
Mikhail Korobov 550d8046fd
Merge pull request #4157 from Gallaecio/flake8 
Add missing whitespace after ‘,’, ‘;’ or ‘:’
 2019-11-14 23:42:06 +05:00
Mikhail Korobov 494f38a99d
Merge pull request #4115 from scrapy/remove-py2-code 
Remove py2 code
 2019-11-14 23:33:58 +05:00
Andrey Rahmatullin 5a2b057355
Merge pull request #4066 from akhterwahab/update-ignored-extensions 
Add .dmg, .iso & .apk to ignored other extensions
 2019-11-14 15:26:03 +05:00
Andrey Rakhmatullin e291460db6 Fix flake8-detected errors. 2019-11-14 15:24:37 +05:00