RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
6,710 Commits 42 Branches 132 Tags 168 MiB
Commit Graph

4175 Commits

Author SHA1 Message Date
Mikhail Korobov c87a4f5c6f remove unused imports from scrapy/settings/__init__.py 
This is a follow-up to https://github.com/scrapy/scrapy/pull/3327
 2018-08-01 01:45:16 +05:00
Eugenio Lacuesta 999341b60b Simplify dupefilter creation 2018-07-26 16:24:58 -03:00
Eugenio Lacuesta 701cd2ff9d Add from_crawler support to dupefilters 2018-07-26 16:24:58 -03:00
Daniel Graña b6abd45926
Merge pull request #3342 from elacuesta/copy-request-flags 
[MRG+1] Include flags when copying requests
 2018-07-25 11:58:25 -03:00
Eugenio Lacuesta 917c1fde68 Merge branch 'master' into enhancement/alternate-feedexport-constructors 2018-07-20 12:09:50 -03:00
Eugenio Lacuesta 98d74d1083 Requested changes 2018-07-20 12:08:49 -03:00
Eugenio Lacuesta 6a38fc39f8 Include flags when copying requests 2018-07-19 11:56:23 -03:00
Daniel Graña 01e317e79b
Merge pull request #3327 from nyov/deprecations 
[MRG+1] Remove deprecated CrawlerSettings class and Settings attributes
 2018-07-11 12:01:49 -03:00
Daniel Graña f8f5f463e6
Merge pull request #3253 from rpkilby/depth-stats 
[MRG+1] Update depth middleware stats (fixes #3245)
 2018-07-11 12:00:08 -03:00
Daniel Graña 9c6d265280
Merge pull request #3152 from nctl144/ftp_linkextractors 
[MRG+1] add ftp to the scheme list
 2018-07-11 11:50:02 -03:00
Konstantin Lopuhin 9428a4a3aa More visible telnet conch message 
Capture traceback when trying to import required twisted modules,
print it in case telnet is enabled, and mention settings variable
that can be used to supress the message.
Thanks @kmike!
 2018-07-09 21:06:51 +03:00
nyov 4f6778aa73 Remove deprecated CrawlerSettings class and Settings attributes 2018-07-09 17:16:31 +00:00
Konstantin Lopuhin f4f39057cb Make csviter work on python 3.7 
PEP 479 does not allow for StopIteration in generators.  Instead,
handle it explicitly, also use a for loop which looks simpler.
 2018-07-09 12:46:45 +03:00
Konstantin Lopuhin 17e9914b8a Catch SyntaxError as well when importing manhole 
Also give a more detailed reason why telnet is not enabled (for the
future).
 2018-07-09 12:26:09 +03:00
Mikhail Korobov f531b66822 SpiderManager shim is removed 2018-07-06 03:28:01 +05:00
Mikhail Korobov 36453348fa remove ancient modules kept only for error messages 2018-07-06 03:23:37 +05:00
Mikhail Korobov d4c7cc848b remove backwards compatibility shims for relocated modules 2018-07-06 03:19:43 +05:00
Oz T 6f5c39d65f Fix for CSV export unnecessary blank lines problem on Windows (#3039) 2018-07-03 18:22:24 -03:00
chainly 74ce156154 add item_error to be catchable (#3256) 2018-07-03 16:00:59 -03:00
Grammy Jiang d05c8677c5 [MRG+1] change the bad smell code (#3304) 
Change the bad smell code
 2018-07-03 15:58:43 -03:00
Vostretsov Nikita 72d0899bce Return non-zero exit code from scrapy commands in case of spider bootstrap errors 
* method to detect spider creation in crawler

* correct method name

* method to know if crawlers has spiders

* we do not need to issue requests

* set exit code accordingly to spiders in crawlers

* more portable way to check ofr exceptions

* more clear way

* test cases for several spiders per crawler

* grammatically correct name for method

* method is private

* grammatically correct name for method

* method is private

* remove unused import

* correct order of imports

* changes mechanism of obtaining spider status from method to object member

* rename tests
 2018-06-14 19:58:48 +05:00
Daniel Graña c6030ce8c6
Merge pull request #3231 from starrify/updating-argument-of-cookiejar-clear 
[MRG+2] Added: Allowing optional arguments for `scrapy.http.cookies.CookieJar.clear`
 2018-06-01 21:52:32 -03:00
Fredrik Bergenlid 6a2d2c3b77 Improve gunzip performance for big files 2018-06-01 21:38:07 +02:00
Ryan P Kilby 6a182c9552 Depth stats are not optional 2018-05-17 11:00:21 -04:00
Ryan P Kilby 2dfc5d128b Update DEPTH_STATS refs to DEPTH_STATS_VERBOSE 2018-05-17 11:00:21 -04:00
Pengyu Chen e75f721c04
Added: Allowing optional arguments for `scrapy.http.cookies.CookieJar.clear` 2018-04-23 22:08:28 +08:00
rhoboro 560ee623fd set defalut value "" to FILES_STORE_GCS_ACL 2018-04-13 19:00:27 +09:00
rhoboro 464973489e Using bucket's default object ACL 2018-04-13 12:06:39 +09:00
rhoboro 5254ac393b added test for gcs policy 2018-04-03 18:06:34 +09:00
rhoboro 8e8994c6b5 add acl support for gcs 2018-04-02 15:36:47 +09:00
Daniel Graña 6c3970e672
Merge pull request #3153 from virmht/new_bug 
[MRG+1] Fixed bug FormRequest.from_response() clickdata ignores input[type=image]
 2018-03-21 16:32:12 -03:00
Viral Mehta dd064413a4 corrected syntax error in XPath 2018-03-19 19:28:41 +05:30
Viral Mehta a5acc9373f Resolving Comments 2018-03-19 18:19:39 +05:30
Viral Mehta e25e2afe17 Removed unnecessary print statements 2018-03-17 18:20:14 +05:30
Viral Mehta ff5f717f7a Fixed formatting issues 2018-03-17 18:17:48 +05:30
Daniel Graña 6cc6bbb5fc
Merge pull request #3166 from lucywang000/catch-tls-certificate-error 
catch CertificateError in tls verification
 2018-03-14 11:47:58 -03:00
Lucy Wang 2c58da19a6 update docstring of ScrapyClientTLSOptions 2018-03-14 09:27:59 +08:00
Lucy Wang 1a2f0193a3 fix tests on jessie 2018-03-13 19:14:52 +08:00
siulkilulki 6a7cdf9a6c [MRG+1] Add 'flv' to ignored video extensions. (#3165) 2018-03-13 10:35:27 +03:00
Lucy Wang 13a74d77e2 catch CertificateError in tls verification 2018-03-12 22:25:19 +08:00
Viral Mehta d5b7ebcfdc Fixed bug FormRequest.from_response() clickdata ignores input[type=image] 2018-03-03 18:17:49 +05:30
nctl144 4c05441450 add ftp to the scheme list 2018-03-03 00:00:03 -05:00
NewUserHa acd2b8d43b [MRG+1] Fix part of issue #3128 - None should not be a valid type for 'url' in Response.follow (#3131) 
* fix one issue of issue#3128

because @kmike posted: 'If url is '', Scrapy should follow the same page, this is an intended behavior.'

*  fix one issue of issue#3128

because @kmike posted: 'If url is '', Scrapy should follow the same page, this is an intended behavior.'
 2018-02-22 03:37:26 +05:00
Daniel Graña c56f7b3c8d
Merge pull request #3113 from WenbinZhang/master 
[MRG+1] Update robotstxt.py
 2018-02-08 18:22:56 -03:00
Daniel Graña 68e45d32e0
Merge pull request #3115 from scrapy/telnet-log-level 
[MRG+1] use INFO log level to show telnet host/port
 2018-02-08 18:22:36 -03:00
Konstantin Lopuhin 936dbc7bf6
Merge branch 'master' into master 2018-02-08 23:43:29 +03:00
Eugenio Lacuesta 6edd4114c4 Clarify comment about Pyhton versions 2018-02-08 15:47:20 -03:00
Eugenio Lacuesta a56540877c Do not serialize unpickable objects (py3) 2018-02-08 15:03:57 -03:00
Konstantin Lopuhin eff469245a
Merge pull request #3100 from scrapy/robots-stats 
[MRG+1] more stats for RobotsTxtMiddleware
 2018-02-08 12:01:36 +03:00
Mikhail Korobov 0c374c00fb use INFO log level to show telnet host/port 2018-02-08 05:09:02 +05:00