RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
6,721 Commits 49 Branches 132 Tags 138 MiB
Commit Graph

6721 Commits

Author SHA1 Message Date
Eugenio Lacuesta 6329441c82 ModuleNotFoundError was added in py3.6 2018-07-18 17:31:31 -03:00
Eugenio Lacuesta c493721b64 Merge branch 'master' into process_spider_exception_generator 2018-07-18 15:18:42 -03:00
Eugenio Lacuesta fa784b1b70 Merge branch 'process_spider_exception_generator_experiment' into process_spider_exception_generator 2018-07-18 15:17:43 -03:00
Eugenio Lacuesta a3af0bfd56 More tests 2018-07-18 15:15:55 -03:00
Mikhail Korobov 258121db04
Merge pull request #3335 from scrapy/docs-retry-codes 
update default RETRY_HTTP_CODES on docs
 2018-07-18 19:33:50 +05:00
Eugenio Lacuesta 610f589662 Add callback and errback in the same step 2018-07-17 19:13:18 -03:00
Eugenio Lacuesta 56e92d90fd Update tests 2018-07-17 15:15:38 -03:00
Eugenio Lacuesta e96b7782bc Merge branch 'master' into process_spider_exception_generator_experiment 2018-07-15 18:17:37 -03:00
Eugenio Lacuesta b8e8922d54 Simplify stuff. Add more tests. 2018-07-15 18:17:00 -03:00
Eugenio Lacuesta 60c2ef86f0 Revert "Default values for OffsiteMiddleware" 
This reverts commit ba29435138.
 2018-07-15 16:47:55 -03:00
Eugenio Lacuesta cff9e87625 Fix tests 2018-07-15 16:21:08 -03:00
Eugenio Lacuesta c5fa0ae6bc Untested experiment 2018-07-14 20:15:53 -03:00
Eugenio Lacuesta 0c579b5276 Untested experiment 2018-07-14 19:58:42 -03:00
Eugenio Lacuesta e7e18db179 Fix tests 2018-07-13 16:40:30 -03:00
Valdir Stumm Junior c61e8a617f
[doc] update default RETRY_HTTP_CODES 2018-07-13 11:55:16 -07:00
Mikhail Korobov c86213317d 1.5.1 release notes 2018-07-12 02:17:08 +05:00
Daniel Graña 01e317e79b
Merge pull request #3327 from nyov/deprecations 
[MRG+1] Remove deprecated CrawlerSettings class and Settings attributes
 2018-07-11 12:01:49 -03:00
Daniel Graña f8f5f463e6
Merge pull request #3253 from rpkilby/depth-stats 
[MRG+1] Update depth middleware stats (fixes #3245)
 2018-07-11 12:00:08 -03:00
Daniel Graña 9c6d265280
Merge pull request #3152 from nctl144/ftp_linkextractors 
[MRG+1] add ftp to the scheme list
 2018-07-11 11:50:02 -03:00
Mikhail Korobov 8d5320dcd2
Merge pull request #3326 from lopuhin/patiences-master 
Python 3.7 support
 2018-07-11 17:36:55 +05:00
Konstantin Lopuhin 9428a4a3aa More visible telnet conch message 
Capture traceback when trying to import required twisted modules,
print it in case telnet is enabled, and mention settings variable
that can be used to supress the message.
Thanks @kmike!
 2018-07-09 21:06:51 +03:00
nyov 4f6778aa73 Remove deprecated CrawlerSettings class and Settings attributes 2018-07-09 17:16:31 +00:00
Konstantin Lopuhin 92b504eae5 Fix telnet warnings in tests 
Disable telnet console if it's not available, else we'll get an extra
warning about failure to enable it, and tests will fail.
 2018-07-09 13:44:02 +03:00
Konstantin Lopuhin b3cd12dc48 Try to get python3.7 by using xenial base and sudo 
See https://github.com/travis-ci/travis-ci/issues/9815#issuecomment-401756442
and https://github.com/travis-ci/travis-ci/issues/9815#issuecomment-402045581
 2018-07-09 13:12:47 +03:00
Konstantin Lopuhin f4f39057cb Make csviter work on python 3.7 
PEP 479 does not allow for StopIteration in generators.  Instead,
handle it explicitly, also use a for loop which looks simpler.
 2018-07-09 12:46:45 +03:00
Konstantin Lopuhin 2773fe09e4 Make "docs" the last build, even though it still uses python3.6 for now 2018-07-09 12:36:58 +03:00
Konstantin Lopuhin cf9399acc1 Use python 3.7 on travis 2018-07-09 12:26:56 +03:00
Konstantin Lopuhin 17e9914b8a Catch SyntaxError as well when importing manhole 
Also give a more detailed reason why telnet is not enabled (for the
future).
 2018-07-09 12:26:09 +03:00
Konstantin Lopuhin 722e1afcdb Update ancient pytest on python 3 
2.9 gives collection errors on python 3.7 due to PEP 479.
 2018-07-09 12:21:19 +03:00
Konstantin Lopuhin 1d2e2735fc Merge branch 'master' of https://github.com/patiences/scrapy into patiences-master 2018-07-09 11:59:22 +03:00
Daniel Graña 666e23714a
Merge pull request #3318 from scrapy/delete-relocation-shims 
[MRG+1] Delete relocation shims
 2018-07-06 13:49:59 -03:00
Mikhail Korobov f531b66822 SpiderManager shim is removed 2018-07-06 03:28:01 +05:00
Mikhail Korobov 36453348fa remove ancient modules kept only for error messages 2018-07-06 03:23:37 +05:00
Mikhail Korobov d4c7cc848b remove backwards compatibility shims for relocated modules 2018-07-06 03:19:43 +05:00
Eugenio Lacuesta 0a0e62272e New tests 2018-07-04 16:19:19 -03:00
Eugenio Lacuesta 0b2870634a Do not inherit from object 2018-07-04 16:14:51 -03:00
Oz T 6f5c39d65f Fix for CSV export unnecessary blank lines problem on Windows (#3039) 2018-07-03 18:22:24 -03:00
chainly 74ce156154 add item_error to be catchable (#3256) 2018-07-03 16:00:59 -03:00
Grammy Jiang d05c8677c5 [MRG+1] change the bad smell code (#3304) 
Change the bad smell code
 2018-07-03 15:58:43 -03:00
Daniel Graña 8d93691a8a
Merge pull request #3311 from scrapy/rename-requirements 
[MRG+1] TST make it clear which requirements are Python 2-only
 2018-07-03 10:33:26 -03:00
Eugenio Lacuesta 985ab636cf Store output methods on the 'methods' dict 2018-07-01 19:11:59 -03:00
Eugenio Lacuesta 4fca9aba85 Recover from a callback exception 2018-07-01 13:45:18 -03:00
Eugenio Lacuesta 6ed9440ed5 Tests for exceptions on spider callbacks 2018-07-01 13:45:18 -03:00
Eugenio Lacuesta 735de8167d Test for exceptions on process_spider_input 2018-07-01 13:45:18 -03:00
Eugenio Lacuesta df75a0942e Update docs 2018-07-01 13:35:14 -03:00
Mikhail Korobov f11d65f7d6 TST make it clear which requirements are Python 2-only 
* rename requirements.txt to requirements-py2.txt, to make it clear they are Python 2-only
* make requirements-py3.txt consistent with requirements-py2.txt
 2018-06-29 18:34:11 +05:00
Daniel Graña 64f48efa26
Merge pull request #3308 from scrapy/fix-build 
TST exclude lxml==4.2.2 from tests
 2018-06-27 23:42:32 -03:00
Mikhail Korobov 8782901fc8
[MRG+1] TST test agains latest pypy (#3309) 
pypy3 is not upgraded, as tests segfault with pypy3 6.0 for some reason
 2018-06-28 01:11:15 +05:00
Mikhail Korobov 45f67eb64d TST exclude lxml==4.2.2 from tests, as it doesn't play well with Pillow 2018-06-28 01:02:30 +05:00
Mikhail Korobov fac1b2f351 TST remove workaround for old Pillow versions which don't support BytesIO 2018-06-28 01:01:27 +05:00