RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
7,401 Commits 42 Branches 132 Tags 168 MiB
Commit Graph

7401 Commits

Author SHA1 Message Date
Eugenio Lacuesta 40449fa0eb Update docs, add tests, remove FIXME comment 2018-08-03 18:20:25 -03:00
Eugenio Lacuesta 8c55f5eb15 Simplify check for re-raised exception. Add tests. 2018-08-03 15:16:26 -03:00
Konstantin Lopuhin eb007bafab
Merge pull request #3359 from scrapy/unused-imports 
remove unused imports from scrapy/settings/__init__.py
 2018-08-02 22:22:40 +03:00
Konstantin Lopuhin d90bdd5d86
Merge pull request #3358 from BurnzZ/fix-error-msg-feedexport-when-using-s3 
[MRG+1] provide better error message when disabling s3 exporter
 2018-08-02 22:21:23 +03:00
Daniel Graña db714f5a07
Merge pull request #3283 from CCInCharge/issue3247 
Fix #3247: Allow scrapy.FormRequest.from_response method to handle duplicate keys
 2018-08-02 14:49:14 -03:00
Mikhail Korobov c87a4f5c6f remove unused imports from scrapy/settings/__init__.py 
This is a follow-up to https://github.com/scrapy/scrapy/pull/3327
 2018-08-01 01:45:16 +05:00
Kevin Lloyd Bernal 980be4cb4b provide better error message when disabling s3 exporter 2018-07-31 19:05:04 +08:00
Eugenio Lacuesta 801d3c07b4 Fix bad exception handling, add tests 2018-07-27 15:06:37 -03:00
Eugenio Lacuesta d6d3e87e3a Rename test file 2018-07-27 14:47:52 -03:00
Daniel Graña 93afe18587
Merge pull request #2956 from elacuesta/dupefilter_from_crawler 
Add from_crawler support to dupefilters
 2018-07-26 17:02:31 -03:00
Eugenio Lacuesta 999341b60b Simplify dupefilter creation 2018-07-26 16:24:58 -03:00
Eugenio Lacuesta 9e14f8c7e4 Fix test for dupefilter 2018-07-26 16:24:58 -03:00
Eugenio Lacuesta 0089a4ab31 Add test for direct creation of dupefilter (no from_crawler/from_settings) 2018-07-26 16:24:58 -03:00
Eugenio Lacuesta d306fe30ac Test dupefilter creation by the Scheduler 2018-07-26 16:24:58 -03:00
Eugenio Lacuesta 701cd2ff9d Add from_crawler support to dupefilters 2018-07-26 16:24:58 -03:00
CCInCharge 782f866572 Fix #3247: Allow scrapy.FormRequest.from_response method to handle duplicate keys 2018-07-25 21:46:42 -07:00
Mikhail Korobov 895df937a3
Merge pull request #3350 from granadoho/master 
make amendments to grammer
 2018-07-25 23:50:06 +03:00
Mikhail Korobov 733ac6327c
Merge pull request #3347 from korigod/doc_copyright-notice 
Doc: update copyright notice
 2018-07-25 23:49:08 +03:00
Daniel Graña b6abd45926
Merge pull request #3342 from elacuesta/copy-request-flags 
[MRG+1] Include flags when copying requests
 2018-07-25 11:58:25 -03:00
Daniel Graña 732d7e1cce
Merge pull request #3348 from elacuesta/enhancement/alternate-feedexport-constructors 
[MRG+1] Add from_crawler constructor for feed exporters and storages
 2018-07-25 11:55:16 -03:00
Malcolm Granado Ho Yong Liang 48866457b3 make amendments to grammer 2018-07-25 14:38:37 +08:00
Eugenio Lacuesta 784eed1130 Improve test coverage (downloader middleware) 2018-07-20 19:08:46 -03:00
Eugenio Lacuesta 917c1fde68 Merge branch 'master' into enhancement/alternate-feedexport-constructors 2018-07-20 12:09:50 -03:00
Eugenio Lacuesta 98d74d1083 Requested changes 2018-07-20 12:08:49 -03:00
Andrei Korigodski 7020c3e452
Doc: update copyright notice 
The years are updated. The hyphen is replaced with an en dash.
 2018-07-20 14:46:57 +03:00
Eugenio Lacuesta 6a38fc39f8 Include flags when copying requests 2018-07-19 11:56:23 -03:00
Eugenio Lacuesta 20defa2e16 Better handling of method indexes 2018-07-19 10:31:06 -03:00
Eugenio Lacuesta 71a1406c99 Logging changes 2018-07-18 17:43:05 -03:00
Eugenio Lacuesta 6329441c82 ModuleNotFoundError was added in py3.6 2018-07-18 17:31:31 -03:00
Eugenio Lacuesta c493721b64 Merge branch 'master' into process_spider_exception_generator 2018-07-18 15:18:42 -03:00
Eugenio Lacuesta fa784b1b70 Merge branch 'process_spider_exception_generator_experiment' into process_spider_exception_generator 2018-07-18 15:17:43 -03:00
Eugenio Lacuesta a3af0bfd56 More tests 2018-07-18 15:15:55 -03:00
Mikhail Korobov 258121db04
Merge pull request #3335 from scrapy/docs-retry-codes 
update default RETRY_HTTP_CODES on docs
 2018-07-18 19:33:50 +05:00
Eugenio Lacuesta 610f589662 Add callback and errback in the same step 2018-07-17 19:13:18 -03:00
Eugenio Lacuesta 56e92d90fd Update tests 2018-07-17 15:15:38 -03:00
Eugenio Lacuesta e96b7782bc Merge branch 'master' into process_spider_exception_generator_experiment 2018-07-15 18:17:37 -03:00
Eugenio Lacuesta b8e8922d54 Simplify stuff. Add more tests. 2018-07-15 18:17:00 -03:00
Eugenio Lacuesta 60c2ef86f0 Revert "Default values for OffsiteMiddleware" 
This reverts commit ba29435138.
 2018-07-15 16:47:55 -03:00
Eugenio Lacuesta cff9e87625 Fix tests 2018-07-15 16:21:08 -03:00
Eugenio Lacuesta c5fa0ae6bc Untested experiment 2018-07-14 20:15:53 -03:00
Eugenio Lacuesta 0c579b5276 Untested experiment 2018-07-14 19:58:42 -03:00
Eugenio Lacuesta e7e18db179 Fix tests 2018-07-13 16:40:30 -03:00
Valdir Stumm Junior c61e8a617f
[doc] update default RETRY_HTTP_CODES 2018-07-13 11:55:16 -07:00
Mikhail Korobov c86213317d 1.5.1 release notes 2018-07-12 02:17:08 +05:00
Daniel Graña 01e317e79b
Merge pull request #3327 from nyov/deprecations 
[MRG+1] Remove deprecated CrawlerSettings class and Settings attributes
 2018-07-11 12:01:49 -03:00
Daniel Graña f8f5f463e6
Merge pull request #3253 from rpkilby/depth-stats 
[MRG+1] Update depth middleware stats (fixes #3245)
 2018-07-11 12:00:08 -03:00
Daniel Graña 9c6d265280
Merge pull request #3152 from nctl144/ftp_linkextractors 
[MRG+1] add ftp to the scheme list
 2018-07-11 11:50:02 -03:00
Mikhail Korobov 8d5320dcd2
Merge pull request #3326 from lopuhin/patiences-master 
Python 3.7 support
 2018-07-11 17:36:55 +05:00
Konstantin Lopuhin 9428a4a3aa More visible telnet conch message 
Capture traceback when trying to import required twisted modules,
print it in case telnet is enabled, and mention settings variable
that can be used to supress the message.
Thanks @kmike!
 2018-07-09 21:06:51 +03:00
nyov 4f6778aa73 Remove deprecated CrawlerSettings class and Settings attributes 2018-07-09 17:16:31 +00:00