From 2f697ce59f8517217da960d7313ce4bab21c2b17 Mon Sep 17 00:00:00 2001 From: Elias Dorneles Date: Wed, 3 Feb 2016 20:51:26 -0200 Subject: [PATCH] added note about how to revert to old behavior for robotstxt --- docs/news.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/news.rst b/docs/news.rst index 0c5228ca2..ea797c1ee 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -32,7 +32,8 @@ This 1.1 release brings a lot of interesting features and bug fixes: If you try ``scrapy shell index.html`` it will try to load the URL http://index.html, use ``scrapy shell ./index.html`` to load a local file. - Robots.txt compliance is now enabled by default for newly-created projects - (:issue:`1724`). + (:issue:`1724`). If you need old behavior, update :setting:`ROBOTSTXT_OBEY` + in ``settings.py`` file when creating a new project. - Exporters now work on unicode, instead of bytes by default (:issue:`1080`). If you use ``PythonItemExporter``, you may want to update your code to disable binary mode which is now deprecated.