RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

added more extensions to ignore on link extractors

This commit is contained in:
Pablo Hoffman 2012-08-07 14:23:50 -03:00
parent 16bee46c70
commit 3bdd9b7b89
1 changed files with 4 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
scrapy/linkextractor.py
View File

@ -16,6 +16,9 @@ IGNORED_EXTENSIONS = [
'3gp', 'asf', 'asx', 'avi', 'mov', 'mp4', 'mpg', 'qt', 'rm', 'swf', 'wmv',
'm4a',
# office suites
'xls', 'ppt', 'doc', 'docx', 'odt', 'ods', 'odg', 'odp',
# other
'css', 'pdf', 'doc', 'exe', 'bin', 'rss', 'zip', 'rar', 'ppt',
'css', 'pdf', 'doc', 'exe', 'bin', 'rss', 'zip', 'rar',
]