RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
4,419 Commits 42 Branches 132 Tags 168 MiB
Commit Graph

3145 Commits

Author SHA1 Message Date
Lev Berman e04b0aff74 An attempt to resolve #977, add signal to be sent when request is dropped by the scheduler 2014-11-27 15:10:15 +03:00
Pablo Hoffman c31fb87335 Merge pull request #954 from kalessin/int-download-timeout 
Force to read DOWNLOAD_TIMEOUT as int (for example to pass using environment variable)
 2014-11-26 17:14:18 -02:00
Martin Olveyra 7910fa0172 Force to read DOWNLOAD_TIMEOUT as int (for example to pass using 
environment variable)
 2014-11-21 01:09:32 -02:00
tpeng a69f042d10 add 2 more test cases and minor doc fixes 2014-11-19 15:31:07 +01:00
tpeng fa84730e70 avoid download large response 
introduce DOWNLOAD_MAXSIZE and DOWNLOAD_WARNSIZE in settings and
download_maxsize/download_warnsize in spider/request meta, so
downloader stop downloading as soon as the received data exceed the
limit. also check the twsisted response's length in advance to stop
downloading as early as possible.
 2014-11-12 12:28:02 +01:00
Pablo Hoffman efe589c643 Merge pull request #882 from ahlen/feature/csvfeed-quotechar 
[MRG+1] Allow to specify the quotechar in CSVFeedSpider
 2014-11-04 11:32:59 -02:00
Nikolaos-Digenis Karagiannis 2227805619 Compatibility with .15 leftover 2014-10-22 21:48:08 +03:00
Pablo Hoffman aa61f615d8 Merge pull request #795 from chekunkov/spider_error_processing_referer 
Add referer to "Spider error processing" log message
 2014-10-07 20:57:48 -02:00
Pablo Hoffman 9af61d5df6 Merge pull request #895 from scrapy/we-are-past-0.15 
[MRG] drop support for CONCURRENT_REQUESTS_PER_SPIDER
 2014-10-03 17:14:00 -03:00
nyov 7db6bbce27 Drop old engine code 
* remove Downloader import unused since 1fba64
  * remove CONCURRENT_SPIDERS deprecation warning from a1dbc6 (2011)
 2014-10-03 19:07:51 +00:00
VKen 33a7c1d438 updated deprecated cgi.parse_qsl to use six's parse_qsl 2014-10-03 04:16:21 +08:00
Pablo Hoffman 5835224eee Merge pull request #896 from scrapy/robotstxt-once 
[MRG] process robots.txt once
 2014-10-02 00:58:55 -03:00
Mikhail Korobov 5086262913 don't hide original exception in scrapy.utils.misc.load_object 2014-09-24 13:27:14 +06:00
Mikhail Korobov 36eec8f413 dont_obey_robotstxt meta key; don't process requests to /robots.txt 2014-09-23 00:10:43 +06:00
Mikhail Korobov fe6f3efe95 RobotsTxtMiddleware: remove unused attribute 2014-09-22 22:56:54 +06:00
Mikhail Korobov d11c8595e6 drop support for CONCURRENT_REQUESTS_PER_SPIDER 2014-09-22 04:29:22 +06:00
Mikhail Korobov 7be3479c20 CookieJar cleanup 2014-09-21 06:37:32 +06:00
Mikhail Korobov c543fe6e4c Merge pull request #878 from andrewshir/master 
Fix bug for ".local" host name
 2014-09-21 05:24:54 +06:00
Mikael Åhlén 47b6dff9f1 Allow to specify the quotechar in CSVFeedSpider 2014-09-13 02:14:57 +02:00
Daniel Graña 5bcabfe9c9 SPIDER_MODULES can be set as a csv string 2014-09-10 23:25:57 -03:00
Daniel Graña a823207f18 Stop logobserver only when set 2014-09-10 12:09:07 -03:00
Daniel Graña ce180227fa Twisted 11.1.0 (precise) can not deal with generators in DeferredList 
Also create a list of the crawlers before iterating them because crawlers are removed from the set once stopped
 2014-09-10 12:04:14 -03:00
Daniel Graña 8ddf0811a8 Correctly detect when all managed crawlers are done in CrawlerRunner 2014-09-09 17:21:39 -03:00
Julia Medina 51532af69a Erase unneeded flag in CrawlerProcess.start 2014-09-07 13:03:34 -03:00
Julia Medina d513b5a542 Run root logger in CrawlerProcess creation instead of in its start method 2014-09-07 13:02:39 -03:00
andrewshir dfca7b3c80 Fix bug for ".local" host name 
It's necessary to put new list member in squared brackets (i.e. create new list) to merge lists properly, otherwise we will get result list with character elements instead of string element.
 2014-09-06 18:23:27 +06:00
eltermann 1dff1fbf75 Removed unused 'load=False' parameter from walk_modules() 2014-09-02 08:33:36 -03:00
Julia Medina 77bd26a66d Non mutable default in Spider.custom_settings 2014-09-01 21:56:57 -03:00
Julia Medina 4932ec43a7 Per-spider settings implementation 2014-09-01 21:56:57 -03:00
Daniel Graña ccde3317d7 Merge pull request #816 from Curita/api-cleanup 
GSoC API cleanup
 2014-09-01 21:55:36 -03:00
Pablo Hoffman 620dbe7116 Merge pull request #865 from yakxxx/master 
SgmlLinkExtractor - fix for parsing <area> tag with Unicode present
 2014-08-29 16:18:44 -03:00
Adam Donahue d92914d297 Fix typo 2014-08-28 20:30:50 -04:00
yakxxx e4689556f0 SgmlLinkExtractor - fix for parsing <area> tag with Unicode present 2014-08-28 18:55:58 +02:00
Mikhail Korobov c4f9e9d8e7 Merge pull request #856 from eliasdorneles/fix-embed-ipython-shell 
fix IPython shell scope issue by using IPython.embed()
 2014-08-20 00:16:33 +06:00
Elias Dorneles 6f50cf55a4 fix IPython shell scope issue and load IPython user config 2014-08-18 18:41:05 -03:00
Daniel Graña d684ecad7b Merge pull request #846 from rocioar/master 
fix dont_merge_cookies bad behaviour when set to false on meta
 2014-08-18 13:54:11 -03:00
Daniel Graña a9292cfab7 jsonrpc webservice moved to https://github.com/scrapy/scrapy-jsonrpc repository 2014-08-15 23:28:13 -03:00
Rocio Aramberri 51b0bd281d fix dont settings on meta behaviour, add docs and tests 2014-08-15 13:47:42 -07:00
Julia Medina 6339864f95 Minor refactor in the docs and functions used in the shell command 2014-08-14 12:32:37 -03:00
Julia Medina 70f2010db1 Change error type when updating frozen settings 2014-08-14 11:59:25 -03:00
Julia Medina 419026615f Deprecate Crawler.spiders attribute 2014-08-14 09:19:41 -03:00
Julia Medina c90977ca98 Drop support for scrapy.project.crawler (And scrapy.stats consequently) 2014-08-12 14:02:56 -03:00
Julia Medina 9cbbfd8b04 Adjust spiders' utils to new SpiderManager API 2014-08-12 14:02:56 -03:00
Julia Medina 900a487682 Support multiple simultaneous LogObservers listening different crawlers 2014-08-12 14:02:56 -03:00
Julia Medina 89df18bd2b Fix usage of crawler_process in ScrapyCommands 2014-08-12 14:02:56 -03:00
Julia Medina d0edad4b0b Drop support for ScrapyCommand.crawler property 2014-08-12 14:02:56 -03:00
Julia Medina 870438e5f4 Update tests utils, fixing get_crawler and removing docrawl 2014-08-12 14:02:56 -03:00
Julia Medina d40273561d CrawlerProcess cleanup changes 2014-08-12 14:02:55 -03:00
Julia Medina 980e30a187 Crawler interface cleanup 2014-08-12 14:02:55 -03:00
Julia Medina d7038b2a13 SpiderManager interface cleanup 2014-08-12 14:02:55 -03:00