RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

allow usage of integer "dont_filter", meaning number of times 

dont_filter will apply when redirecting.

Under this scheme, dont_filter=True is the same as dont_filter=1
and (convenient?) dont filter <= 0 means dont_filter = True forever

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40143
This commit is contained in:
olveyra 2008-08-01 18:13:07 +00:00
parent 3badb39ed7
commit 7a195a9794
1 changed files with 4 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
scrapy/trunk/scrapy/contrib/downloadermiddleware/redirect.py
View File

@ -25,7 +25,10 @@ class RedirectMiddleware(object):
redirected.body = None
# This is needed to avoid redirection loops with requests that contain dont_filter = True
# Example (9 May 2008): http://www.55max.com/product/001_photography.asp?3233,0,0,0,Michael+Banks
redirected.dont_filter = False
if isinstance(redirected.dont_filter, int):
redirected.dont_filter -= 1
else:
redirected.dont_filter = False
log.msg("Redirecting (%s) to %s from %s" % (status, redirected, request), level=log.DEBUG, domain=spider.domain_name)
return redirected
log.msg("Ignored redirecting (%s) to %s from %s (disabled by spider)" % (status, redirected_url, request), level=log.DEBUG, domain=spider.domain_name)