RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
scrapy/scrapy
History
tpeng fa84730e70 avoid download large response 
introduce DOWNLOAD_MAXSIZE and DOWNLOAD_WARNSIZE in settings and
download_maxsize/download_warnsize in spider/request meta, so
downloader stop downloading as soon as the received data exceed the
limit. also check the twsisted response's length in advance to stop
downloading as early as possible.
 		2014-11-12 12:28:02 +01:00
..
commands Correctly detect when all managed crawlers are done in CrawlerRunner 2014-09-09 17:21:39 -03:00
contracts Fix usage of crawler_process in ScrapyCommands 2014-08-12 14:02:56 -03:00
contrib Merge pull request #882 from ahlen/feature/csvfeed-quotechar 2014-11-04 11:32:59 -02:00
contrib_exp use io.BytesIO and cStringIO instead of six.BytesIO as suggested 2014-07-15 20:51:12 +08:00
core avoid download large response 2014-11-12 12:28:02 +01:00
http CookieJar cleanup 2014-09-21 06:37:32 +06:00
selector suggest scrapy.Selector in deprecation warnings 2014-04-15 01:09:35 +06:00
settings avoid download large response 2014-11-12 12:28:02 +01:00
templates add utf8 encoding header to spider templates 2014-06-26 00:30:27 +06:00
utils Merge pull request #882 from ahlen/feature/csvfeed-quotechar 2014-11-04 11:32:59 -02:00
xlib PY3: use six.iterkeys, six.iteritems, and tempfile 2014-07-14 18:47:22 +08:00
VERSION Bump version: 0.25.0 → 0.25.1 2014-07-09 03:05:39 -03:00
__init__.py PY3 top-level shortcuts work 2014-08-02 00:24:32 +06:00
_monkeypatches.py cleanup toplevel namespace 2014-04-15 01:09:35 +06:00
cmdline.py PY3: use six.iterkeys, six.iteritems, and tempfile 2014-07-14 18:47:22 +08:00
command.py Drop support for ScrapyCommand.crawler property 2014-08-12 14:02:56 -03:00
conf.py better backwards compatibility for scrapy.conf.settings 2012-10-25 11:41:40 -02:00
crawler.py Stop logobserver only when set 2014-09-10 12:09:07 -03:00
dupefilter.py rename attribute to match conventions used for XXX_DEBUG settings (in autothrottle and cookies mw) 2014-02-20 12:03:06 -02:00
exceptions.py pylint cleanup: unused imports and old-style exceptions 2013-10-18 19:23:45 +06:00
extension.py Made ExtensionManager a subclass of MiddlewareManager 2010-08-22 05:33:08 -03:00
interfaces.py SpiderManager interface cleanup 2014-08-12 14:02:55 -03:00
item.py items: Use BaseItem.__hash__ and six.add_metaclass decorator 2014-07-24 15:37:10 -03:00
link.py PY3 port scrapy.link 2014-08-02 00:16:01 +06:00
linkextractor.py PY3: use six for robotparser and urlparse 2014-07-14 21:26:37 +08:00
log.py Support multiple simultaneous LogObservers listening different crawlers 2014-08-12 14:02:56 -03:00
logformatter.py Log level return from LogFormatter methods 2012-12-29 11:43:41 +07:00
mail.py PY3: use six.BytesIO and six.moves.cStringIO 2014-07-15 15:52:10 +08:00
middleware.py Promote startup info on settings and middleware to INFO level 2014-01-09 19:37:06 +01:00
mime.types moved module scrapy.core.downloader.responsetypes to scrapy.responsetypes 2011-08-07 02:49:57 -03:00
project.py Drop support for scrapy.project.crawler (And scrapy.stats consequently) 2014-08-12 14:02:56 -03:00
resolver.py remove unused imports and some assorted pylint-ing 2013-08-23 13:03:34 -03:00
responsetypes.py PY3 MimeTypes wants files in text mode. 2014-08-01 03:28:05 +06:00
shell.py Drop support for scrapy.project.crawler (And scrapy.stats consequently) 2014-08-12 14:02:56 -03:00
signalmanager.py remove debugging code 2013-02-27 03:52:55 -02:00
signals.py removed request_received and added request_scheduled 2013-06-26 16:45:46 -03:00
spider.py Non mutable default in Spider.custom_settings 2014-09-01 21:56:57 -03:00
spidermanager.py SPIDER_MODULES can be set as a csv string 2014-09-10 23:25:57 -03:00
squeue.py PY3: use six.moves.cPickle 2014-07-14 17:15:28 +08:00
stats.py Drop support for scrapy.project.crawler (And scrapy.stats consequently) 2014-08-12 14:02:56 -03:00
statscol.py minor message update: spider stats -> Scrapy stats 2012-10-12 16:53:44 -02:00
telnet.py telnet client: fix unexisting reference to engine.slots 2013-11-19 04:52:24 +01:00