diff --git a/docs/news.rst b/docs/news.rst index 900c6d1c5..25f459a49 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -3,107 +3,109 @@ Release notes ============= -1.1.0 (unreleased) ------------------- +1.1.0 +----- -Python 3 Support (basic) -~~~~~~~~~~~~~~~~~~~~~~~~ +This 1.1 release brings a lot of interesting features and bug fixes: + +- Scrapy 1.1 has basic Python 3 support (requires Twisted >= 15.5). See + :ref:`news_basicpy3` for more details and some limitations. +- Hot new features: + + - ItemLoaders now support nested loaders (:issue:`1467`). + - ``FormRequest.from_response`` improvements (:issue:`1382`, :issue:`1137`). + - Added setting :setting:`AUTOTHROTTLE_TARGET_CONCURRENCY` and improved + AutoThrottle docs (:issue:`1324`). + - Added ``response.text`` to get body as unicode (:issue:`1730`). + - Anonymous S3 connections (:issue:`1358`). + - Deferreds in downloader middlewares (:issue:`1473`). This enables better + robots.txt handling (:issue:`1471`). + - HTTP cache improvements (:issue:`1151`). + +- These bug fixes may require your attention: + + - Don't retry bad requests (HTTP 400) (:issue:`1289`). + - Fix shell files argument handling (:issue:`1710`, :issue:`1550`). + - Fixes on robots.txt handling (:issue:`1783`). + - Exporters work on unicode (:issue:`1080`). + - Fix xmliter to accept nodenames with dots (:issue:`1533`). + +Keep reading for more details on other improvements and bug fixes. + +.. _news_basicpy3: + +Basic Python 3 Support +~~~~~~~~~~~~~~~~~~~~~~ We have been hard at work to make Scrapy run on Python 3. As a result, now you -can run spiders on Python 3.3, 3.4 and 3.5, although some features are still -missing (some of them may never be ported to Python 3). +can run spiders on Python 3.3, 3.4 and 3.5 (Twisted >= 15.5 required). Some +features are still missing (and some may never be ported). Almost all addons/middlewares are expected to work. However, we are aware of some limitations: -- s3 downloads are not supported (see :issue:`1718`) -- sending emails is not supported +- S3 downloads are not supported (see :issue:`1718`) +- Sending emails is not supported - FTP download handler is not supported (non-Python 3 ported Twisted dependency) -- telnet is not supported (non-Python 3 ported Twisted dependency) -- there are problems with non-ASCII URLs in Python 3 -- reported problems with HTTP caches created by Scrapy in Python 2.x which +- Telnet is not supported (non-Python 3 ported Twisted dependency) +- There are problems with non-ASCII URLs in Python 3 +- Reported problems with HTTP caches created by Scrapy in Python 2.x which can't be reused in Scrapy in Python 3.x (to be checked) -- there is also a nasty issue with `cryptography` library: recent versions +- There is also a nasty issue with `cryptography` library: recent versions don't work well on OS X + Python 3.5 (see https://github.com/pyca/cryptography/issues/2690). As a workaround, you can downgrade the library to an older version. -New Features and Enhancements -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Additional New Features and Enhancements +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Scrapy now has a `Code of Conduct`_ (:issue:`1681`). - Command line tool now has completion for zsh (:issue:`934`). -- ``scrapy shell`` got a few changes of its own: +- Improvements to ``scrapy shell``: - - it now checks a new ``SCRAPY_PYTHON_SHELL`` environment - variable to launch the interactive shell of your choice - (one of ``ipython``, ``bpython`` or ``python`` that is); - - it will try ``bpython`` if ``ipython`` is not available - (:issue:`1444`). - - it uses ``http://`` as the default scheme for URLs (:issue:`1498`) - (try ``scrapy shell scrapy.org``) + - Support for bpython and configure preferred Python shell via + ``SCRAPY_PYTHON_SHELL`` (:issue:`1100`, :issue:`1444`). + - Support URLs without scheme (:issue:`1498`) **Warning: backwards incompatible!** - - unless argument looks like a relative file path, which works again; - this was a regression identified in 1.0+ releases - (:issue:`1710`, :issue:`1550`). + - Bring back support for relative file path (:issue:`1710`, :issue:`1550`). -- Autothrottle code has been cleaned up and its docs have been improved; - there's also a new ``AUTOTHROTTLE_TARGET_CONCURRENCY`` setting which - allows to send more than 1 concurrent request on average (:issue:`1324`). -- Memory usage extension got a new ``MEMUSAGE_CHECK_INTERVAL_SECONDS`` - setting to change default check interval (:issue:`1282`). -- HTTP caching now follows RFC2616 more closely (TODO: link to docs); - 2 new settings can be used to control level of compliancy: - ``HTTPCACHE_ALWAYS_STORE`` and ``HTTPCACHE_IGNORE_RESPONSE_CACHE_CONTROLS`` - (:issue:`1151`). +- Added :setting:`MEMUSAGE_CHECK_INTERVAL_SECONDS` setting to change default check + interval (:issue:`1282`). +- HTTP caching now follows RFC2616 more closely, added settings + :setting:`HTTPCACHE_ALWAYS_STORE` and + :setting:`HTTPCACHE_IGNORE_RESPONSE_CACHE_CONTROLS` (:issue:`1151`). - Download handlers are now lazy-loaded on first request using their scheme (:issue:`1390`, :issue:`1421`). -- RedirectMiddleware now skips the status codes from - ``handle_httpstatus_list``. You can set it either as a spider attribute - or in ``Request``'s ``meta`` key (:issue:`1334`, :issue:`1364`, :issue:`1447`). +- ``RedirectMiddleware`` now skips the status codes from + ``handle_httpstatus_list`` on spider attribute + or in ``Request``'s ``meta`` key (:issue:`1334`, :issue:`1364`, + :issue:`1447`). - Form submission: - now works with ``