RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Fail-fast path. 

For me middleware now can process about 2-3k ajax crawlable pages/sec and 50k+ regular pages/sec (if they don’t contain «fragment» or «content» words).
This commit is contained in:
Mikhail Korobov 2013-12-19 00:28:47 +06:00
parent 71c59e1c94
commit 503ab58f59
1 changed files with 10 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

10
scrapy/contrib/downloadermiddleware/ajaxcrawlable.py
View File

@ -75,7 +75,17 @@ def _has_ajaxcrawlable_meta(text):
>>> _has_ajaxcrawlable_meta('<html></html>')
False
"""
# Stripping scripts and comments is slow (about 20x slower than
# just checking if a string is in text); this is a quick fail-fast
# path that should work for most pages.
if 'fragment' not in text:
return False
if 'content' not in text:
return False
text = _script_re.sub(u'', text)
text = _noscript_re.sub(u'', text)
text = html.remove_comments(html.remove_entities(text))
return _ajax_crawlable_re.search(text) is not None