RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
9,291 Commits 49 Branches 132 Tags 155 MiB
Commit Graph

9291 Commits

Author SHA1 Message Date
Eugenio Lacuesta d306fe30ac Test dupefilter creation by the Scheduler 2018-07-26 16:24:58 -03:00
Eugenio Lacuesta 701cd2ff9d Add from_crawler support to dupefilters 2018-07-26 16:24:58 -03:00
CCInCharge 782f866572 Fix #3247: Allow scrapy.FormRequest.from_response method to handle duplicate keys 2018-07-25 21:46:42 -07:00
Mikhail Korobov 895df937a3
Merge pull request #3350 from granadoho/master 
make amendments to grammer
 2018-07-25 23:50:06 +03:00
Mikhail Korobov 733ac6327c
Merge pull request #3347 from korigod/doc_copyright-notice 
Doc: update copyright notice
 2018-07-25 23:49:08 +03:00
Daniel Graña b6abd45926
Merge pull request #3342 from elacuesta/copy-request-flags 
[MRG+1] Include flags when copying requests
 2018-07-25 11:58:25 -03:00
Daniel Graña 732d7e1cce
Merge pull request #3348 from elacuesta/enhancement/alternate-feedexport-constructors 
[MRG+1] Add from_crawler constructor for feed exporters and storages
 2018-07-25 11:55:16 -03:00
Malcolm Granado Ho Yong Liang 48866457b3 make amendments to grammer 2018-07-25 14:38:37 +08:00
Eugenio Lacuesta 784eed1130 Improve test coverage (downloader middleware) 2018-07-20 19:08:46 -03:00
Eugenio Lacuesta 917c1fde68 Merge branch 'master' into enhancement/alternate-feedexport-constructors 2018-07-20 12:09:50 -03:00
Eugenio Lacuesta 98d74d1083 Requested changes 2018-07-20 12:08:49 -03:00
Andrei Korigodski 7020c3e452
Doc: update copyright notice 
The years are updated. The hyphen is replaced with an en dash.
 2018-07-20 14:46:57 +03:00
Eugenio Lacuesta 6a38fc39f8 Include flags when copying requests 2018-07-19 11:56:23 -03:00
Eugenio Lacuesta 20defa2e16 Better handling of method indexes 2018-07-19 10:31:06 -03:00
Eugenio Lacuesta 71a1406c99 Logging changes 2018-07-18 17:43:05 -03:00
Eugenio Lacuesta 6329441c82 ModuleNotFoundError was added in py3.6 2018-07-18 17:31:31 -03:00
Eugenio Lacuesta c493721b64 Merge branch 'master' into process_spider_exception_generator 2018-07-18 15:18:42 -03:00
Eugenio Lacuesta fa784b1b70 Merge branch 'process_spider_exception_generator_experiment' into process_spider_exception_generator 2018-07-18 15:17:43 -03:00
Eugenio Lacuesta a3af0bfd56 More tests 2018-07-18 15:15:55 -03:00
Mikhail Korobov 258121db04
Merge pull request #3335 from scrapy/docs-retry-codes 
update default RETRY_HTTP_CODES on docs
 2018-07-18 19:33:50 +05:00
Eugenio Lacuesta 610f589662 Add callback and errback in the same step 2018-07-17 19:13:18 -03:00
Eugenio Lacuesta 56e92d90fd Update tests 2018-07-17 15:15:38 -03:00
Eugenio Lacuesta e96b7782bc Merge branch 'master' into process_spider_exception_generator_experiment 2018-07-15 18:17:37 -03:00
Eugenio Lacuesta b8e8922d54 Simplify stuff. Add more tests. 2018-07-15 18:17:00 -03:00
Eugenio Lacuesta 60c2ef86f0 Revert "Default values for OffsiteMiddleware" 
This reverts commit ba29435138.
 2018-07-15 16:47:55 -03:00
Eugenio Lacuesta cff9e87625 Fix tests 2018-07-15 16:21:08 -03:00
Eugenio Lacuesta c5fa0ae6bc Untested experiment 2018-07-14 20:15:53 -03:00
Eugenio Lacuesta 0c579b5276 Untested experiment 2018-07-14 19:58:42 -03:00
Eugenio Lacuesta e7e18db179 Fix tests 2018-07-13 16:40:30 -03:00
Valdir Stumm Junior c61e8a617f
[doc] update default RETRY_HTTP_CODES 2018-07-13 11:55:16 -07:00
Mikhail Korobov c86213317d 1.5.1 release notes 2018-07-12 02:17:08 +05:00
Daniel Graña 01e317e79b
Merge pull request #3327 from nyov/deprecations 
[MRG+1] Remove deprecated CrawlerSettings class and Settings attributes
 2018-07-11 12:01:49 -03:00
Daniel Graña f8f5f463e6
Merge pull request #3253 from rpkilby/depth-stats 
[MRG+1] Update depth middleware stats (fixes #3245)
 2018-07-11 12:00:08 -03:00
Daniel Graña 9c6d265280
Merge pull request #3152 from nctl144/ftp_linkextractors 
[MRG+1] add ftp to the scheme list
 2018-07-11 11:50:02 -03:00
Mikhail Korobov 8d5320dcd2
Merge pull request #3326 from lopuhin/patiences-master 
Python 3.7 support
 2018-07-11 17:36:55 +05:00
Konstantin Lopuhin 9428a4a3aa More visible telnet conch message 
Capture traceback when trying to import required twisted modules,
print it in case telnet is enabled, and mention settings variable
that can be used to supress the message.
Thanks @kmike!
 2018-07-09 21:06:51 +03:00
nyov 4f6778aa73 Remove deprecated CrawlerSettings class and Settings attributes 2018-07-09 17:16:31 +00:00
Konstantin Lopuhin 92b504eae5 Fix telnet warnings in tests 
Disable telnet console if it's not available, else we'll get an extra
warning about failure to enable it, and tests will fail.
 2018-07-09 13:44:02 +03:00
Konstantin Lopuhin b3cd12dc48 Try to get python3.7 by using xenial base and sudo 
See https://github.com/travis-ci/travis-ci/issues/9815#issuecomment-401756442
and https://github.com/travis-ci/travis-ci/issues/9815#issuecomment-402045581
 2018-07-09 13:12:47 +03:00
Konstantin Lopuhin f4f39057cb Make csviter work on python 3.7 
PEP 479 does not allow for StopIteration in generators.  Instead,
handle it explicitly, also use a for loop which looks simpler.
 2018-07-09 12:46:45 +03:00
Konstantin Lopuhin 2773fe09e4 Make "docs" the last build, even though it still uses python3.6 for now 2018-07-09 12:36:58 +03:00
Konstantin Lopuhin cf9399acc1 Use python 3.7 on travis 2018-07-09 12:26:56 +03:00
Konstantin Lopuhin 17e9914b8a Catch SyntaxError as well when importing manhole 
Also give a more detailed reason why telnet is not enabled (for the
future).
 2018-07-09 12:26:09 +03:00
Konstantin Lopuhin 722e1afcdb Update ancient pytest on python 3 
2.9 gives collection errors on python 3.7 due to PEP 479.
 2018-07-09 12:21:19 +03:00
Konstantin Lopuhin 1d2e2735fc Merge branch 'master' of https://github.com/patiences/scrapy into patiences-master 2018-07-09 11:59:22 +03:00
Daniel Graña 666e23714a
Merge pull request #3318 from scrapy/delete-relocation-shims 
[MRG+1] Delete relocation shims
 2018-07-06 13:49:59 -03:00
Mikhail Korobov f531b66822 SpiderManager shim is removed 2018-07-06 03:28:01 +05:00
Mikhail Korobov 36453348fa remove ancient modules kept only for error messages 2018-07-06 03:23:37 +05:00
Mikhail Korobov d4c7cc848b remove backwards compatibility shims for relocated modules 2018-07-06 03:19:43 +05:00
Eugenio Lacuesta 0a0e62272e New tests 2018-07-04 16:19:19 -03:00