RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

minor adjustment to FifoDomainScheduler and improved documentation of domain scheduler API (remove_pending_domain method removes all ocurrences)

This commit is contained in:
Pablo Hoffman 2009-06-19 19:41:56 -03:00
parent 161335fe78
commit 728ec7c5c9
1 changed files with 7 additions and 6 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

13
scrapy/contrib/domainsch.py
View File

@ -1,18 +1,19 @@
"""
Domain Schedulers keep track of next domains to scrape. They must implement the
following methods:
The Domain Scheduler keeps track of next domains to scrape. They must implement
the following methods:
* next_domain()
return next domain to scrape and remove it from pending queue
* add_domain(domain)
add domain to pending domains to scrape
add domain to pending queue
* remove_pending_domain(domain)
remove domain from pendings, do nothing if not pending
remove (all occurrences) of domain from pending queue, do nothing if not
pending
* has_pending_domain(domain)
Return ``True`` if the domain is pending, ``False`` otherwise
Return ``True`` if the domain is pending to scrape, ``False`` otherwise
"""
@ -30,7 +31,7 @@ class FifoDomainScheduler(object):
self.pending_domains.append(domain)
def remove_pending_domain(self, domain):
self.pending_domains.remove(domain)
self.pending_domains = [d for d in self.pending_domains if d != domain]
def has_pending_domain(self, domain):
return domain in self.pending_domains