Mikhail Korobov f30758c246 Enable robots.txt handling by default for new projects. Fixes GH-1668. ... For backwards compatibility reasons the default value is not changed.		2016-01-26 17:47:46 +05:00
..
project	Enable robots.txt handling by default for new projects. Fixes GH-1668.	2016-01-26 17:47:46 +05:00
spiders	Move scrapy/contrib/spiders to scrapy/spiders	2015-04-29 21:27:19 -03:00