RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

run without RobotTxtMiddleware by default and update docs 

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40796
This commit is contained in:
Daniel Grana 2009-01-29 18:22:59 +00:00
parent e6375c6a5b
commit f4e6ed3c2f
2 changed files with 2 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
scrapy/trunk/docs/ref/settings.rst
View File

@ -656,7 +656,7 @@ can be scheduled per domain.
ROBOTSTXT_OBEY
--------------
Default: ``True``
Default: ``False``
Scope: ``scrapy.contrib.downloadermiddleware.robotstxt``

2
scrapy/trunk/scrapy/conf/default_settings.py
View File

@ -143,7 +143,7 @@ REQUESTS_PER_DOMAIN = 8 # max simultaneous requests per domain
RETRY_TIMES = 2 # initial response + 2 retries = 3 requests
RETRY_HTTP_CODES = ['500', '503', '504', '400', '408', '200']
ROBOTSTXT_OBEY = True
ROBOTSTXT_OBEY = False
SCHEDULER = 'scrapy.core.scheduler.Scheduler'
SCHEDULER_ORDER = 'BFO' # available orders: BFO (default), DFO