diff --git a/docs/news.rst b/docs/news.rst index 0e9efe6f6..8647041f6 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -3,8 +3,77 @@ Release notes ============= -1.1.3 (2016-09-22) ------------------- +Scrapy 1.2.0 (2016-10-03) +------------------------- + +New Features +~~~~~~~~~~~~ + +- New :setting:`FEED_EXPORT_ENCODING` setting to customize the encoding + used when writing items to a file. + This can be used to turn off ``\uXXXX`` escapes in JSON output. + This is also useful for those wanting something else than UTF-8 + for XML or CSV output (:issue:`2034`). +- ``startproject`` command now supports an optional destination directory + to override the default one based on the project name (:issue:`2005`). +- New :setting:`SCHEDULER_DEBUG` setting to log requests serialization + failures (:issue:`1610`). +- JSON encoder now supports serialization of ``set`` instances (:issue:`2058`). +- Interpret ``application/json-amazonui-streaming`` as ``TextResponse`` (:issue:`1503`). +- ``scrapy`` is imported by default when using shell tools (:command:`shell`, + :ref:`inspect_response `) (:issue:`2248`). + +Bug fixes +~~~~~~~~~ + +- DefaultRequestHeaders middleware now runs before UserAgent middleware + (:issue:`2088`). **Warning: this is technically backwards incompatible**, + though we consider this a bug fix. +- HTTP cache extension and plugins that use the ``.scrapy`` data directory now + work outside projects (:issue:`1581`). **Warning: this is technically + backwards incompatible**, though we consider this a bug fix. +- ``Selector`` does not allow passing both ``response`` and ``text`` anymore + (:issue:`2153`). +- Fixed logging of wrong callback name with ``scrapy parse`` (:issue:`2169`). +- Fix for an odd gzip decompression bug (:issue:`1606`). +- Fix for selected callbacks when using ``CrawlSpider`` with :command:`scrapy parse ` + (:issue:`2225`). +- Fix for invalid JSON and XML files when spider yields no items (:issue:`872`). +- Implement ``flush()`` fpr ``StreamLogger`` avoiding a warning in logs (:issue:`2125`). + +Refactoring +~~~~~~~~~~~ + +- ``canonicalize_url`` has been moved to `w3lib.url`_ (:issue:`2168`). + +.. _w3lib.url: http://w3lib.readthedocs.io/en/latest/w3lib.html#w3lib.url.canonicalize_url + +Tests & Requirements +~~~~~~~~~~~~~~~~~~~~ + +Scrapy's new requirements baseline is Debian 8 "Jessie". It was previously +Ubuntu 12.04 Precise. +What this means in practice is that we run continuous integration tests +with these (main) packages versions at a minimum: +Twisted 14.0, pyOpenSSL 0.14, lxml 3.4. + +Scrapy may very well work with older versions of these packages +(the code base still has switches for older Twisted versions for example) +but it is not guaranteed (because it's not tested anymore). + +Documentation +~~~~~~~~~~~~~ + +- Grammar fixes: :issue:`2128`, :issue:`1566`. +- Download stats badge removed from README (:issue:`2160`). +- New scrapy :ref:`architecture diagram ` (:issue:`2165`). +- Updated ``Response`` parameters documentation (:issue:`2197`). +- Reworded misleading :setting:`RANDOMIZE_DOWNLOAD_DELAY` description (:issue:`2190`). +- Add StackOverflow as a support channel (:issue:`2257`). + + +Scrapy 1.1.3 (2016-09-22) +------------------------- Bug fixes ~~~~~~~~~ @@ -20,8 +89,8 @@ Documentation (:issue:`2236`, :issue:`2249`, :issue:`2252`). -1.1.2 (2016-08-18) ------------------- +Scrapy 1.1.2 (2016-08-18) +------------------------- Bug fixes ~~~~~~~~~ @@ -34,8 +103,8 @@ Bug fixes (the regression was introduced in 1.1.1) -1.1.1 (2016-07-13) ------------------- +Scrapy 1.1.1 (2016-07-13) +------------------------- Bug fixes ~~~~~~~~~ @@ -86,8 +155,8 @@ Tests - Upgrade py.test requirement on Travis CI and Pin pytest-cov to 2.2.1 (:issue:`2095`) -1.1.0 (2016-05-11) ------------------- +Scrapy 1.1.0 (2016-05-11) +------------------------- This 1.1 release brings a lot of interesting features and bug fixes: @@ -275,8 +344,8 @@ Bugfixes to same remote host (:issue:`1912`). -1.0.6 (2016-05-04) ------------------- +Scrapy 1.0.6 (2016-05-04) +------------------------- - FIX: RetryMiddleware is now robust to non-standard HTTP status codes (:issue:`1857`) - FIX: Filestorage HTTP cache was checking wrong modified time (:issue:`1875`) @@ -284,8 +353,8 @@ Bugfixes - DOC: Consistency in selectors examples (:issue:`1869`) -1.0.5 (2016-02-04) ------------------- +Scrapy 1.0.5 (2016-02-04) +------------------------- - FIX: [Backport] Ignore bogus links in LinkExtractors (fixes :issue:`907`, :commit:`108195e`) - TST: Changed buildbot makefile to use 'pytest' (:commit:`1f3d90a`) @@ -293,8 +362,8 @@ Bugfixes - DOC: Add AjaxCrawlMiddleware to DOWNLOADER_MIDDLEWARES_BASE in settings docs (:commit:`aa94121`) -1.0.4 (2015-12-30) ------------------- +Scrapy 1.0.4 (2015-12-30) +------------------------- - Ignoring xlib/tx folder, depending on Twisted version. (:commit:`7dfa979`) - Run on new travis-ci infra (:commit:`6e42f0b`) @@ -345,14 +414,14 @@ Bugfixes - Small grammatical change (:commit:`8752294`) - Add openssl version to version command (:commit:`13c45ac`) -1.0.3 (2015-08-11) ------------------- +Scrapy 1.0.3 (2015-08-11) +------------------------- - add service_identity to scrapy install_requires (:commit:`cbc2501`) - Workaround for travis#296 (:commit:`66af9cd`) -1.0.2 (2015-08-06) ------------------- +Scrapy 1.0.2 (2015-08-06) +------------------------- - Twisted 15.3.0 does not raises PicklingError serializing lambda functions (:commit:`b04dd7d`) - Minor method name fix (:commit:`6f85c7f`) @@ -361,8 +430,8 @@ Bugfixes - Fixed typos (:commit:`a9ae7b0`) - Fix doc reference. (:commit:`7c8a4fe`) -1.0.1 (2015-07-01) ------------------- +Scrapy 1.0.1 (2015-07-01) +------------------------- - Unquote request path before passing to FTPClient, it already escape paths (:commit:`cc00ad2`) - include tests/ to source distribution in MANIFEST.in (:commit:`eca227e`) @@ -371,8 +440,8 @@ Bugfixes - DOC remove version suffix from ubuntu package (:commit:`5303c66`) - DOC Update release date for 1.0 (:commit:`c89fa29`) -1.0.0 (2015-06-19) ------------------- +Scrapy 1.0.0 (2015-06-19) +------------------------- You will find a lot of new features and bugfixes in this major release. Make sure to check our updated :ref:`overview ` to get a glance of @@ -737,8 +806,8 @@ Code refactoring (:issue:`805`) - rename "sflo" local variables to less cryptic "log_observer" (:issue:`775`) -0.24.6 (2015-04-20) -------------------- +Scrapy 0.24.6 (2015-04-20) +-------------------------- - encode invalid xpath with unicode_escape under PY2 (:commit:`07cb3e5`) - fix IPython shell scope issue and load IPython user config (:commit:`2c8e573`) @@ -747,8 +816,8 @@ Code refactoring - Converted sel.xpath() calls to response.xpath() in Extracting the data (:commit:`c2c6d15`) -0.24.5 (2015-02-25) -------------------- +Scrapy 0.24.5 (2015-02-25) +-------------------------- - Support new _getEndpoint Agent signatures on Twisted 15.0.0 (:commit:`540b9bc`) - DOC a couple more references are fixed (:commit:`b4c454b`) @@ -773,14 +842,14 @@ Code refactoring - Update request-response.rst (:commit:`3f3263d`) - SgmlLinkExtractor - fix for parsing tag with Unicode present (:commit:`49b40f0`) -0.24.4 (2014-08-09) -------------------- +Scrapy 0.24.4 (2014-08-09) +-------------------------- - pem file is used by mockserver and required by scrapy bench (:commit:`5eddc68`) - scrapy bench needs scrapy.tests* (:commit:`d6cb999`) -0.24.3 (2014-08-09) -------------------- +Scrapy 0.24.3 (2014-08-09) +-------------------------- - no need to waste travis-ci time on py3 for 0.24 (:commit:`8e080c1`) - Update installation docs (:commit:`1d0c096`) @@ -812,23 +881,23 @@ Code refactoring - better testcase for settings.overrides.setdefault (:commit:`e22daaf`) - Using CRLF as line marker according to http 1.1 definition (:commit:`5ec430b`) -0.24.2 (2014-07-08) -------------------- +Scrapy 0.24.2 (2014-07-08) +-------------------------- - Use a mutable mapping to proxy deprecated settings.overrides and settings.defaults attribute (:commit:`e5e8133`) - there is not support for python3 yet (:commit:`3cd6146`) - Update python compatible version set to debian packages (:commit:`fa5d76b`) - DOC fix formatting in release notes (:commit:`c6a9e20`) -0.24.1 (2014-06-27) -------------------- +Scrapy 0.24.1 (2014-06-27) +-------------------------- - Fix deprecated CrawlerSettings and increase backwards compatibility with .defaults attribute (:commit:`8e3f20a`) -0.24.0 (2014-06-26) -------------------- +Scrapy 0.24.0 (2014-06-26) +-------------------------- Enhancements ~~~~~~~~~~~~ @@ -907,15 +976,15 @@ Bugfixes - Testsuite doesn't require PIL anymore (:issue:`585`) -0.22.2 (released 2014-02-14) ----------------------------- +Scrapy 0.22.2 (released 2014-02-14) +----------------------------------- - fix a reference to unexistent engine.slots. closes #593 (:commit:`13c099a`) - downloaderMW doc typo (spiderMW doc copy remnant) (:commit:`8ae11bf`) - Correct typos (:commit:`1346037`) -0.22.1 (released 2014-02-08) ----------------------------- +Scrapy 0.22.1 (released 2014-02-08) +----------------------------------- - localhost666 can resolve under certain circumstances (:commit:`2ec2279`) - test inspect.stack failure (:commit:`cc3eda3`) @@ -943,8 +1012,8 @@ Bugfixes - fix 0.22.0 release date (:commit:`af0219a`) - fix typos in news.rst and remove (not released yet) header (:commit:`b7f58f4`) -0.22.0 (released 2014-01-17) ----------------------------- +Scrapy 0.22.0 (released 2014-01-17) +----------------------------------- Enhancements ~~~~~~~~~~~~ @@ -987,20 +1056,20 @@ Fixes - Fix tests runner under pip 1.5 (:issue:`513`) - Fix logging error when spider name is unicode (:issue:`479`) -0.20.2 (released 2013-12-09) ----------------------------- +Scrapy 0.20.2 (released 2013-12-09) +----------------------------------- - Update CrawlSpider Template with Selector changes (:commit:`6d1457d`) - fix method name in tutorial. closes GH-480 (:commit:`b4fc359` -0.20.1 (released 2013-11-28) ----------------------------- +Scrapy 0.20.1 (released 2013-11-28) +----------------------------------- - include_package_data is required to build wheels from published sources (:commit:`5ba1ad5`) - process_parallel was leaking the failures on its internal deferreds. closes #458 (:commit:`419a780`) -0.20.0 (released 2013-11-08) ----------------------------- +Scrapy 0.20.0 (released 2013-11-08) +----------------------------------- Enhancements ~~~~~~~~~~~~ @@ -1087,15 +1156,15 @@ List of contributors sorted by number of commits:: 1 cacovsky 1 Berend Iwema -0.18.4 (released 2013-10-10) ----------------------------- +Scrapy 0.18.4 (released 2013-10-10) +----------------------------------- - IPython refuses to update the namespace. fix #396 (:commit:`3d32c4f`) - Fix AlreadyCalledError replacing a request in shell command. closes #407 (:commit:`b1d8919`) - Fix start_requests laziness and early hangs (:commit:`89faf52`) -0.18.3 (released 2013-10-03) ----------------------------- +Scrapy 0.18.3 (released 2013-10-03) +----------------------------------- - fix regression on lazy evaluation of start requests (:commit:`12693a5`) - forms: do not submit reset inputs (:commit:`e429f63`) @@ -1103,14 +1172,14 @@ List of contributors sorted by number of commits:: - backport master fixes to json exporter (:commit:`cfc2d46`) - Fix permission and set umask before generating sdist tarball (:commit:`06149e0`) -0.18.2 (released 2013-09-03) ----------------------------- +Scrapy 0.18.2 (released 2013-09-03) +----------------------------------- - Backport `scrapy check` command fixes and backward compatible multi crawler process(:issue:`339`) -0.18.1 (released 2013-08-27) ----------------------------- +Scrapy 0.18.1 (released 2013-08-27) +----------------------------------- - remove extra import added by cherry picked changes (:commit:`d20304e`) - fix crawling tests under twisted pre 11.0.0 (:commit:`1994f38`) @@ -1128,8 +1197,8 @@ List of contributors sorted by number of commits:: - minor updates to 0.18 release notes (:commit:`c45e5f1`) - fix contributters list format (:commit:`0b60031`) -0.18.0 (released 2013-08-09) ----------------------------- +Scrapy 0.18.0 (released 2013-08-09) +----------------------------------- - Lot of improvements to testsuite run using Tox, including a way to test on pypi - Handle GET parameters for AJAX crawleable urls (:commit:`3fe2a32`) @@ -1220,8 +1289,8 @@ contributors sorted by number of commits:: 1 Berend Iwema -0.16.5 (released 2013-05-30) ----------------------------- +Scrapy 0.16.5 (released 2013-05-30) +----------------------------------- - obey request method when scrapy deploy is redirected to a new endpoint (:commit:`8c4fcee`) - fix inaccurate downloader middleware documentation. refs #280 (:commit:`40667cb`) @@ -1229,8 +1298,8 @@ contributors sorted by number of commits:: - Find form nodes in invalid html5 documents (:commit:`e3d6945`) - Fix typo labeling attrs type bool instead of list (:commit:`a274276`) -0.16.4 (released 2013-01-23) ----------------------------- +Scrapy 0.16.4 (released 2013-01-23) +----------------------------------- - fixes spelling errors in documentation (:commit:`6d2b3aa`) - add doc about disabling an extension. refs #132 (:commit:`c90de33`) @@ -1241,8 +1310,8 @@ contributors sorted by number of commits:: - fix bug in scrapy parse command when spider is not specified explicitly. closes #209 (:commit:`c72e682`) - Update docs/topics/commands.rst (:commit:`28eac7a`) -0.16.3 (released 2012-12-07) ----------------------------- +Scrapy 0.16.3 (released 2012-12-07) +----------------------------------- - Remove concurrency limitation when using download delays and still ensure inter-request delays are enforced (:commit:`487b9b5`) - add error details when image pipeline fails (:commit:`8232569`) @@ -1254,8 +1323,8 @@ contributors sorted by number of commits:: - Fixed docs typo in SpiderOpenCloseLogging example (:commit:`7184094`) -0.16.2 (released 2012-11-09) ----------------------------- +Scrapy 0.16.2 (released 2012-11-09) +----------------------------------- - scrapy contracts: python2.6 compat (:commit:`a4a9199`) - scrapy contracts verbose option (:commit:`ec41673`) @@ -1265,8 +1334,8 @@ contributors sorted by number of commits:: - Fix SpiderState bug in Windows platforms (:commit:`58998f4`) -0.16.1 (released 2012-10-26) ----------------------------- +Scrapy 0.16.1 (released 2012-10-26) +----------------------------------- - fixed LogStats extension, which got broken after a wrong merge before the 0.16 release (:commit:`8c780fd`) - better backwards compatibility for scrapy.conf.settings (:commit:`3403089`) @@ -1276,8 +1345,8 @@ contributors sorted by number of commits:: - set release date for 0.16.0 in news (:commit:`e292246`) -0.16.0 (released 2012-10-18) ----------------------------- +Scrapy 0.16.0 (released 2012-10-18) +----------------------------------- Scrapy changes: @@ -1318,8 +1387,8 @@ Scrapy changes: - number received responses are now tracked through Scrapy stats (stat name: ``response_received_count``) - removed ``scrapy.log.started`` attribute -0.14.4 ------- +Scrapy 0.14.4 +------------- - added precise to supported ubuntu distros (:commit:`b7e46df`) - fixed bug in json-rpc webservice reported in https://groups.google.com/forum/#!topic/scrapy-users/qgVBmFybNAQ/discussion. also removed no longer supported 'run' command from extras/scrapy-ws.py (:commit:`340fbdb`) @@ -1327,8 +1396,8 @@ Scrapy changes: - replace "import Image" by more standard "from PIL import Image". closes #88 (:commit:`4d17048`) - return trial status as bin/runtests.sh exit value. #118 (:commit:`b7b2e7f`) -0.14.3 ------- +Scrapy 0.14.3 +------------- - forgot to include pydispatch license. #118 (:commit:`fd85f9c`) - include egg files used by testsuite in source distribution. #118 (:commit:`c897793`) @@ -1340,8 +1409,8 @@ Scrapy changes: - fixed minor defect in link extractors documentation (:commit:`ba14f38`) - removed some obsolete remaining code related to sqlite support in scrapy (:commit:`0665175`) -0.14.2 ------- +Scrapy 0.14.2 +------------- - move buffer pointing to start of file before computing checksum. refs #92 (:commit:`6a5bef2`) - Compute image checksum before persisting images. closes #92 (:commit:`9817df1`) @@ -1355,8 +1424,8 @@ Scrapy changes: - scrapyd: fixed documentation link (:commit:`2b4e4c3`) - extras/makedeb.py: no longer obtaining version from git (:commit:`caffe0e`) -0.14.1 ------- +Scrapy 0.14.1 +------------- - extras/makedeb.py: no longer obtaining version from git (:commit:`caffe0e`) - bumped version to 0.14.1 (:commit:`6cb9e1c`) @@ -1370,8 +1439,8 @@ Scrapy changes: - Avoid _disconnectedDeferred AttributeError exception in Twisted>=11.1.0 (:commit:`98f3f87`) - allow spider to set autothrottle max concurrency (:commit:`175a4b5`) -0.14 ----- +Scrapy 0.14 +----------- New features and settings ~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -1438,8 +1507,8 @@ Code rearranged and removed - Renamed attributes of core components: downloader.sites -> downloader.slots, scraper.sites -> scraper.slots (:rev:`2717`, :rev:`2718`) - Renamed setting ``CLOSESPIDER_ITEMPASSED`` to :setting:`CLOSESPIDER_ITEMCOUNT` (:rev:`2655`). Backwards compatibility kept. -0.12 ----- +Scrapy 0.12 +----------- The numbers like #NNN reference tickets in the old issue tracker (Trac) which is no longer available. @@ -1481,8 +1550,8 @@ Deprecated/obsoleted functionality - Deprecated ``queue`` command in favor of using Scrapyd ``schedule.json`` API. See also: Scrapyd changes - Removed the !LxmlItemLoader (experimental contrib which never graduated to main contrib) -0.10 ----- +Scrapy 0.10 +----------- The numbers like #NNN reference tickets in the old issue tracker (Trac) which is no longer available. @@ -1554,8 +1623,8 @@ Changes to settings - Removed ``COMMANDS_SETTINGS_MODULE`` setting (#201) - Renamed ``REQUEST_HANDLERS`` to ``DOWNLOAD_HANDLERS`` and make download handlers classes (instead of functions) -0.9 ---- +Scrapy 0.9 +---------- The numbers like #NNN reference tickets in the old issue tracker (Trac) which is no longer available. @@ -1594,8 +1663,8 @@ Changes to default settings - Changed default ``SCHEDULER_ORDER`` to ``DFO`` (:rev:`1939`) -0.8 ---- +Scrapy 0.8 +---------- The numbers like #NNN reference tickets in the old issue tracker (Trac) which is no longer available. @@ -1640,8 +1709,8 @@ Backwards-incompatible changes - Renamed extension: ``DelayedCloseDomain`` to ``SpiderCloseDelay`` (:rev:`1861` | #121) - Removed obsolete ``scrapy.utils.markup.remove_escape_chars`` function - use ``scrapy.utils.markup.replace_escape_chars`` instead (:rev:`1865`) -0.7 ---- +Scrapy 0.7 +---------- First release of Scrapy.