RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
7,017 Commits 49 Branches 132 Tags 144 MiB
Commit Graph

887 Commits

Author SHA1 Message Date
Maram Sumanth aa46e1995c [MRG+1] Show elapsed time in statscollector (#3638) 
* Update corestats.py

* Update corestats.py

* corrected tests

* Update corestats.py

* Update scrapy/extensions/corestats.py
 2019-04-07 00:03:40 +05:00
Kostis Anagnostopoulos 3a493b6066 fix: do not catch system exceptions like KeyboardInterrupt 2019-04-05 11:52:00 +02:00
Konstantin Lopuhin b5c552d17f
Merge pull request #2061 from elacuesta/process_spider_exception_generator 
[MRG+1] process_spider_exception on generators
 2019-04-01 10:43:19 +03:00
Mikhail Korobov ec719f55e7
Merge pull request #3682 from elacuesta/rule_process_request_response_parameter 
[MRG+1] Rule.process_request: access Response object
 2019-03-28 01:39:54 +05:00
Maram Sumanth 2fd8b7c28c [MRG+1] redirect_reasons in Request.meta (#3687) 2019-03-27 00:15:53 +05:00
Eugenio Lacuesta 1b4385b7e3 Rule.process_request: move deprecation warnings and compiling code, update tests 2019-03-22 19:46:17 -03:00
Mikhail Korobov 729580dae2
Merge pull request #3671 from Gallaecio/item-deepcopy 
Implement Item.deepcopy()
 2019-03-22 22:56:39 +05:00
Adrián Chaves 7c148fce5a Implement Item.deepcopy() 2019-03-22 17:07:10 +01:00
Konstantin Lopuhin af2b666595
Merge pull request #3505 from kasun/json_request 
[MRG+2] Request subclass for json requests #3504
 2019-03-22 18:25:54 +03:00
Eugenio Lacuesta 01ed605d02 PEP8 changes to test_spider.py 2019-03-15 16:54:14 +00:00
Victor Torres fda1d04b81
Merge branch 'master' into feed-storage-s3-acl 2019-03-15 13:26:45 -03:00
Adrián Chaves 70a0f1f2e4
Merge pull request #3610 from Gallaecio/spidercls-check 
[MRG+1] Check that spidercls arguments in scrapy.crawler classes are not spid…
 2019-03-15 08:25:16 +01:00
Adrián Chaves 1f7413dc88
Merge pull request #3635 from matthieucham/feature-filteringlinkextractor-restrict-text 
[MRG+1] Feature filteringlinkextractor restrict text
 2019-03-15 07:58:28 +01:00
Mikhail Korobov 5dc94db847
Merge branch 'master' into backward 2019-03-14 22:21:09 +05:00
Eugenio Lacuesta 22fda61d62 Rule.process_request: tests 2019-03-13 10:54:38 +00:00
Adrián Chaves 75d6f56c8a Switch from ` to `` where inline code formatting is desired 2019-03-01 16:56:58 +01:00
Adrián Chaves 858f5be747 backwards → backward (adj.) 2019-03-01 16:10:23 +01:00
Matthieu Grandrie e3b15252c8 New constructor arg *restrict_text* for FilteringLinkExtractor. 
Same as allow and deny args, it holds a string, a regex or an iterable of. Links whose text don't match one of the regex are filtered out.
DOC restrict_text in LxmlLinkExtractor
 2019-02-28 17:21:17 +01:00
Victor Torres 9b8ba4c383 try to import botocore before runing some tests 2019-02-14 16:20:56 -02:00
Victor Torres ea8be627d1 botocore is not supported on debian jessie 2019-02-13 20:32:36 -02:00
Victor Torres dc0b643832 refactoring tests to avoid mocking private method 2019-02-13 19:44:50 -02:00
Adrián Chaves 03e61b9908 Check that spidercls arguments in scrapy.crawler classes are not spider objects 2019-02-12 14:46:27 +01:00
Victor Torres 1eac2a163c simplifying how we deal with threads.deferToThread calls 2019-02-08 16:50:39 -02:00
Victor Torres f824f5b2d1 testing public method store instead of private method _store_in_thread 
need to mock deferToThread function
 2019-02-08 15:51:23 -02:00
Victor Torres 079af889e7 also testing without botocore 2019-02-08 15:51:23 -02:00
Victor Torres 126207fb7b PEP8: use short name for mock method 2019-02-08 15:51:23 -02:00
Victor Torres ad83ffdf1f refactoring 2019-02-08 15:51:23 -02:00
Daniel Graña 65d631329a Be consistent with domain used for links to documentation website 2019-01-31 01:28:53 -03:00
Júlio César Batista 8eade7d864 Testing stats and log messages from RFPDupeFilter 2019-01-18 11:39:35 -02:00
kasun Herath f3813e376c Merge remote-tracking branch 'origin/master' into json_request 2019-01-14 23:03:21 +05:30
kasun Herath 3f914f6d8c made jsonrequest dump into private method 2019-01-14 23:03:14 +05:30
Eugenio Lacuesta 9759112a3a Merge branch 'master' into process_spider_exception_generator 2019-01-03 11:34:07 -03:00
Victor Torres fe283bcd05 add test case for sitemap filter with alternate links 2018-12-27 11:40:14 -03:00
Victor Torres 5e7ecf9dc1 add tests for sitemapindex 2018-12-27 11:40:14 -03:00
Victor Torres 657f0663b3 rename param from urls to entries 2018-12-27 11:40:14 -03:00
Victor Torres d7d5917ff1 add tests for the sitemap_filter method in the SitemapSpider class 2018-12-27 11:40:14 -03:00
Daniel Graña 93cf38354d
Merge pull request #3394 from hcoura/dh-lazyloading-optional 
Make lazy loading Download Handlers optional
 2018-12-26 11:16:19 -03:00
Daniel Graña f6ce716372
Merge pull request #3476 from elacuesta/deque_appendleft 
[MRG+1] Use collections.deque instead of list to store MiddlewareManager's methods
 2018-12-26 11:13:52 -03:00
Mikhail Korobov dbfabf02e8 Merge branch 'master' into telnet-auth 2018-12-26 01:04:34 +05:00
kasun Herath 8f1507a4a5 dumps_kwargs 2018-12-17 23:14:06 +05:30
kasun Herath ecda69130e allow to send empty data values and docs changes 2018-12-10 22:34:49 +05:30
kasun Herath c347acbff6 warning if body and data are provided 2018-12-09 11:27:09 +05:30
kasun Herath cd619c1d4f removed overriden replace method 2018-12-08 22:10:45 +05:30
kasun Herath 1b2b8b4bf0 fix tests under py3 2018-11-27 08:57:44 +05:30
kasun Herath 1ce6662a9d Implement Request subclass for json requests 2018-11-24 20:02:00 +05:30
Eugenio Lacuesta f97e3e90f2 Use collections.deque instead of list to store methods 2018-10-29 12:40:20 -03:00
Eugenio Lacuesta e0360e5223 Add tests for MutableChain 2018-10-11 11:55:13 -03:00
Eugenio Lacuesta 2396356a82 Merge branch 'master' into process_spider_exception_generator 2018-10-10 11:37:43 -03:00
Henrique Coura 441e1e750f Style changes 2018-09-26 13:28:34 -03:00
Henrique Coura e57a629efc Generate only password, encode username/password only on login 2018-09-26 11:54:57 -03:00