RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

html entities regexp improvement 

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40274
This commit is contained in:
Damian Canabal 2008-09-24 12:25:10 +00:00
parent 79be2ca97d
commit 4e7e539b03
1 changed files with 3 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
scrapy/trunk/scrapy/utils/markup.py
View File

@ -5,7 +5,7 @@ Functions for dealing with markup text
import re
import htmlentitydefs
_ent_re = re.compile(r'&(#?)(.+?);')
_ent_re = re.compile(r'&(#?)([^&;]+);')
_tag_re = re.compile(r'<[a-zA-Z\/!].*?>', re.DOTALL)
def remove_entities(text, keep=(), remove_illegal=True):
@ -15,9 +15,9 @@ def remove_entities(text, keep=(), remove_illegal=True):
If 'keep' is passed (with a list of entity names) those entities will
be kept (they won't be removed).
It supports both numeric (&#nnnn;) and named (&nbsp; &gt;) entities.
If remove_illegal is True, entities that can't be converted are removed.
If remove_illegal is False, entities that can't be converted are kept "as
is". For more information see the tests.