From 851adcedf2220a783cdfb2d6b5872580e18b8d88 Mon Sep 17 00:00:00 2001 From: Paul Tremberth Date: Mon, 6 Mar 2017 23:36:21 +0100 Subject: [PATCH 01/12] List merged pull requests since 1.3.3 --- docs/news.rst | 73 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 73 insertions(+) diff --git a/docs/news.rst b/docs/news.rst index da856d883..675a6d595 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -3,6 +3,78 @@ Release notes ============= +Scrapy 1.4.0 (2017-XX-XX) +------------------------- + +New Features +~~~~~~~~~~~~ + +- Use credentials from request.meta['proxy'] #2530 +- [httpcompression] add support for br - brotli content encoding #2535 +- Enable memusage extension by default. #2539 +- response.follow #2540 +- add flags to request #2082 +- Support Anonymous FTP #2343 +- HttpErrorMiddleware stats #2566 +- Retry stats #2543 +- Set canonicalize=False for LinkExtractor #2537 +- Referrer policies in RefererMiddleware #2306 +- Fix referrer policy from response headers and support explicit empty string #2627 +- Data URI download handler. #2334 +- HttpCacheMiddleware: log cache directory at instantiation #2611 +- Add warning on duplicate spider name #2612 +- Allowed passing objects of Mapping class or its subclass to the CaselessDict initializer #2646 +- Allow redirections in media files downloads #2616 +- Travis CI: use portable pypy for Linux #2710 + + +Bug fixes +~~~~~~~~~ + +- LinkExtractors: strip whitespaces #2547 +- FormRequest: handle whitespaces in action attribute properly #2548 +- Buffer CONNECT response bytes from proxy until all HTTP headers are received #2495 +- Fix FTP downloader and re-enable FTP tests on Python 3 #2599 +- Handle data loss gracefully. #2590 +- Use body to choose response type after decompression content #2393 +- Always decompress Content-Encoding: gzip at HttpCompression stage #2391 +- Respect custom log level (#2581, fixes #1612) +- [logformatter] 'flags' format spec backward compatibility #2649 +- 'make htmlview' does not open the webbrowser #2661 +- Remove "commands" from the command list #2695 + +Cleanups +~~~~~~~~ + +- TST remove temp files and folders #2570 +- TST fixed ProjectUtilsTest on OS X #2569 +- Separate building request from _requests_to_follow in CrawlSpider #2562 +- remove “Python 3 progress” badge #2567 +- add a couple more lines to gitignore #2557 +- deprecate Spider.make_requests_from_url. #1728 +- Remove bumpversion prerelease configuration #2159 +- Set context factory implementation based on Twisted version #2577 +- Add omitted "self" arguments #2595 +- Remove redundant slot.add_request() call in ExecutionEngine #2617 +- Removed contrib section in contribution documentation #2636 +- More specific exception catching: os.path.getmtime can only raise os.error in FSFilesStore #2644 + +Documentation +~~~~~~~~~~~~~ + +- Doc: binary mode is required for exporters #2564 +- document issue with FormRequest.from_response due to bug in lxml #2572 +- Use single quotes uniformly #2596 +- Document ftp_user and ftp_password meta keys #2587 +- Update release notes for 1.0.7, 1.1.4 and 1.2.3 #2625 +- DOC Mention brotli support in HttpCompressionMiddleware section #2628 +- Removed contrib section in contribution documentation #2636 +- docs: installation instructions, mention conda in the beginning (closes #2475) #2477 +- FAQ Rewrite note on Python 3 support on Windows #2690 +- DOC Rearrange selector sections #2705 +- Remove __nonzero__ from SelectorList docs #2683 + + Scrapy 1.3.3 (2017-03-10) ------------------------- @@ -15,6 +87,7 @@ Bug fixes A new setting is introduced to toggle between warning or exception if needed ; see :setting:`SPIDER_LOADER_WARN_ONLY` for details. + Scrapy 1.3.2 (2017-02-13) ------------------------- From 8729a91f7a136decb66b00ada858a67c41796e11 Mon Sep 17 00:00:00 2001 From: Paul Tremberth Date: Wed, 19 Apr 2017 17:22:37 +0200 Subject: [PATCH 02/12] Rephrase "New features" section --- docs/news.rst | 50 +++++++++++++++++++++----------- docs/topics/media-pipeline.rst | 5 ++-- docs/topics/request-response.rst | 2 +- 3 files changed, 36 insertions(+), 21 deletions(-) diff --git a/docs/news.rst b/docs/news.rst index 675a6d595..979e9a953 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -9,24 +9,40 @@ Scrapy 1.4.0 (2017-XX-XX) New Features ~~~~~~~~~~~~ -- Use credentials from request.meta['proxy'] #2530 -- [httpcompression] add support for br - brotli content encoding #2535 -- Enable memusage extension by default. #2539 -- response.follow #2540 -- add flags to request #2082 -- Support Anonymous FTP #2343 -- HttpErrorMiddleware stats #2566 -- Retry stats #2543 -- Set canonicalize=False for LinkExtractor #2537 -- Referrer policies in RefererMiddleware #2306 -- Fix referrer policy from response headers and support explicit empty string #2627 -- Data URI download handler. #2334 -- HttpCacheMiddleware: log cache directory at instantiation #2611 -- Add warning on duplicate spider name #2612 -- Allowed passing objects of Mapping class or its subclass to the CaselessDict initializer #2646 -- Allow redirections in media files downloads #2616 -- Travis CI: use portable pypy for Linux #2710 +- Accept proxy credentials in request.meta['proxy'] (:issue:`2526`) +- Support `brotli`_-compressed content; requires optional `brotlipy`_ + (:issue:`2535`) +- Enable memusage extension by default (:issue:`2187`) ; + **this is technically backwards-incompatible** so please check if you have + any non-default ``MEMUSAGE_***`` settings set. +- New :meth:`Response.follow ` shortcur + for creating requests (:issue:`1940`) +- Added ``flags`` argument and attribute to :class:`Request ` + (:issue:`2047`) +- Support Anonymous FTP (:issue:`2342`) +- Added ``retry/count``, ``retry/max_reached`` and ``retry/reason_count/***`` + stats to :class:`RetryMiddleware ` + (:issue:`2543`) +- Added ``httperror/response_ignored_count`` and ``httperror/response_ignored_status_count/***`` + stats to :class:`HttpErrorMiddleware ` + (:issue:`2566`) +- Default to ``canonicalize=False`` in :class:`scrapy.linkextractors.LinkExtractor` + (:issue:`2537`, fixes :issue:`1941` and :issue:`1982`): + **warning, this istechnically backwards-incompatible** +- Customizable :setting:`Referrer policy ` in + :class:`RefererMiddleware ` + (:issue:`2306`) +- New ``data:`` URI download handler (:issue:`2334`, fixes :issue:`2156`) +- Log cache directory when HTTP Cache is used (:issue:`2611`, fixes :issue:`2604`) +- Warn users when project contains duplicate spider names (fixes :issue:`2181`) +- :class:`CaselessDict` now accepts ``Mapping`` instances and not only dicts (:issue:`2646`) +- :ref:`Media downloads `, with :class:`FilesPipelines` + or :class:`ImagesPipelines`, can now optionally handle HTTP redirects + using the new :setting:`MEDIA_ALLOW_REDIRECTS` (:issue:`2616`, fixes :issue:`2004`) +- Use portable pypy for Linux on Travis CI (:issue:`2710`) +.. _brotli: https://github.com/google/brotli +.. _brotlipy: https://github.com/python-hyper/brotlipy/ Bug fixes ~~~~~~~~~ diff --git a/docs/topics/media-pipeline.rst b/docs/topics/media-pipeline.rst index f258ff748..e948913a4 100644 --- a/docs/topics/media-pipeline.rst +++ b/docs/topics/media-pipeline.rst @@ -320,8 +320,6 @@ all be dropped because at least one dimension is shorter than the constraint. By default, there are no size constraints, so all images are processed. -.. _topics-media-pipeline-override: - Allowing redirections --------------------- @@ -330,10 +328,11 @@ Allowing redirections By default media pipelines ignore redirects, i.e. an HTTP redirection to a media file URL request will mean the media download is considered failed. -To handle media redirections, set this settings to ``True``: +To handle media redirections, set this setting to ``True``:: MEDIA_ALLOW_REDIRECTS = True +.. _topics-media-pipeline-override: Extending the Media Pipelines ============================= diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rst index f1552572a..6ca37b7c9 100644 --- a/docs/topics/request-response.rst +++ b/docs/topics/request-response.rst @@ -24,7 +24,7 @@ below in :ref:`topics-request-response-ref-request-subclasses` and Request objects =============== -.. class:: Request(url[, callback, method='GET', headers, body, cookies, meta, encoding='utf-8', priority=0, dont_filter=False, errback]) +.. class:: Request(url[, callback, method='GET', headers, body, cookies, meta, encoding='utf-8', priority=0, dont_filter=False, errback, flags]) A :class:`Request` object represents an HTTP request, which is usually generated in the Spider and executed by the Downloader, and thus generating From cba55cd190c573a03b8458cda7ba2a20da5651e6 Mon Sep 17 00:00:00 2001 From: Paul Tremberth Date: Fri, 21 Apr 2017 16:52:32 +0200 Subject: [PATCH 03/12] Rephrase other sections --- docs/news.rst | 96 ++++++++++++++++++++++++++++----------------------- 1 file changed, 52 insertions(+), 44 deletions(-) diff --git a/docs/news.rst b/docs/news.rst index 979e9a953..a19d7a93a 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -9,26 +9,26 @@ Scrapy 1.4.0 (2017-XX-XX) New Features ~~~~~~~~~~~~ -- Accept proxy credentials in request.meta['proxy'] (:issue:`2526`) +- Accept proxy credentials in :reqmeta:`proxy` request meta key (:issue:`2526`) - Support `brotli`_-compressed content; requires optional `brotlipy`_ (:issue:`2535`) - Enable memusage extension by default (:issue:`2187`) ; **this is technically backwards-incompatible** so please check if you have any non-default ``MEMUSAGE_***`` settings set. -- New :meth:`Response.follow ` shortcur +- New :meth:`Response.follow ` shortcut for creating requests (:issue:`1940`) - Added ``flags`` argument and attribute to :class:`Request ` - (:issue:`2047`) + objects (:issue:`2047`) - Support Anonymous FTP (:issue:`2342`) -- Added ``retry/count``, ``retry/max_reached`` and ``retry/reason_count/***`` +- Added ``retry/count``, ``retry/max_reached`` and ``retry/reason_count/`` stats to :class:`RetryMiddleware ` (:issue:`2543`) -- Added ``httperror/response_ignored_count`` and ``httperror/response_ignored_status_count/***`` +- Added ``httperror/response_ignored_count`` and ``httperror/response_ignored_status_count/`` stats to :class:`HttpErrorMiddleware ` (:issue:`2566`) - Default to ``canonicalize=False`` in :class:`scrapy.linkextractors.LinkExtractor` (:issue:`2537`, fixes :issue:`1941` and :issue:`1982`): - **warning, this istechnically backwards-incompatible** + **warning, this is technically backwards-incompatible** - Customizable :setting:`Referrer policy ` in :class:`RefererMiddleware ` (:issue:`2306`) @@ -38,8 +38,9 @@ New Features - :class:`CaselessDict` now accepts ``Mapping`` instances and not only dicts (:issue:`2646`) - :ref:`Media downloads `, with :class:`FilesPipelines` or :class:`ImagesPipelines`, can now optionally handle HTTP redirects - using the new :setting:`MEDIA_ALLOW_REDIRECTS` (:issue:`2616`, fixes :issue:`2004`) -- Use portable pypy for Linux on Travis CI (:issue:`2710`) + using the new :setting:`MEDIA_ALLOW_REDIRECTS` setting (:issue:`2616`, fixes :issue:`2004`) +- Accept non-complete responses from websites using a new + :setting:`DOWNLOAD_FAIL_ON_DATALOSS` setting (:issue:`2590`, fixes :issue:`2586`) .. _brotli: https://github.com/google/brotli .. _brotlipy: https://github.com/python-hyper/brotlipy/ @@ -47,48 +48,55 @@ New Features Bug fixes ~~~~~~~~~ -- LinkExtractors: strip whitespaces #2547 -- FormRequest: handle whitespaces in action attribute properly #2548 -- Buffer CONNECT response bytes from proxy until all HTTP headers are received #2495 -- Fix FTP downloader and re-enable FTP tests on Python 3 #2599 -- Handle data loss gracefully. #2590 -- Use body to choose response type after decompression content #2393 -- Always decompress Content-Encoding: gzip at HttpCompression stage #2391 -- Respect custom log level (#2581, fixes #1612) -- [logformatter] 'flags' format spec backward compatibility #2649 -- 'make htmlview' does not open the webbrowser #2661 -- Remove "commands" from the command list #2695 +- LinkExtractor now strips leading and trailing whitespaces from attributes + (:issue:`2547`, fixes :issue:`1614`) +- Properly handle whitespaces in action attribute in :class:`FormRequest` + (:issue:`2548`) +- Buffer CONNECT response bytes from proxy until all HTTP headers are received + (:issue:`2495`, fixes :issue:`2491`) +- FTP downloader now works on Python 3, provided you use Twisted>=17.1 + (:issue:`2599`) +- Use body to choose response type after decompressing content (:issue:`2393`, + fixes :issue:`2145`) +- Always decompress ``Content-Encoding: gzip`` at :class:`HttpCompressionMiddleware + ` stage (:issue:`2391`) +- Respect custom log level in ``Spider.custom_settings`` (:issue:`2581`, + fixes :issue:`1612`) +- 'make htmlview' fix for macOS (:issue:`2661`) +- Remove "commands" from the command list (:issue:`2695`) -Cleanups -~~~~~~~~ +Cleanups & Refactoring +~~~~~~~~~~~~~~~~~~~~~~ -- TST remove temp files and folders #2570 -- TST fixed ProjectUtilsTest on OS X #2569 -- Separate building request from _requests_to_follow in CrawlSpider #2562 -- remove “Python 3 progress” badge #2567 -- add a couple more lines to gitignore #2557 -- deprecate Spider.make_requests_from_url. #1728 -- Remove bumpversion prerelease configuration #2159 -- Set context factory implementation based on Twisted version #2577 -- Add omitted "self" arguments #2595 -- Remove redundant slot.add_request() call in ExecutionEngine #2617 -- Removed contrib section in contribution documentation #2636 -- More specific exception catching: os.path.getmtime can only raise os.error in FSFilesStore #2644 +- Tests: remove temp files and folders (:issue:`2570`), + fixed ProjectUtilsTest on OS X (:issue:`2569`), + use portable pypy for Linux on Travis CI (:issue:`2710`) + +- Separate building request from ``_requests_to_follow`` in CrawlSpider (:issue:`2562`) +- Remove “Python 3 progress” badge (:issue:`2567`) +- Add a couple more lines to ``.gitignore`` (:issue:`2557`) +- Deprecate ``Spider.make_requests_from_url`` (:issue:`1728`) +- Remove bumpversion prerelease configuration (:issue:`2159`) +- Set context factory implementation based on Twisted version (:issue:`2577`, + fixes :issue:`2560`) +- Add omitted ``self`` arguments in default project middleware template (:issue:`2595`) +- Remove redundant ``slot.add_request()`` call in ExecutionEngine (:issue:`2617`) +- Catch more specific ``os.error`` exception in :class:`FSFilesStore` (:issue:`2644`) Documentation ~~~~~~~~~~~~~ -- Doc: binary mode is required for exporters #2564 -- document issue with FormRequest.from_response due to bug in lxml #2572 -- Use single quotes uniformly #2596 -- Document ftp_user and ftp_password meta keys #2587 -- Update release notes for 1.0.7, 1.1.4 and 1.2.3 #2625 -- DOC Mention brotli support in HttpCompressionMiddleware section #2628 -- Removed contrib section in contribution documentation #2636 -- docs: installation instructions, mention conda in the beginning (closes #2475) #2477 -- FAQ Rewrite note on Python 3 support on Windows #2690 -- DOC Rearrange selector sections #2705 -- Remove __nonzero__ from SelectorList docs #2683 +- Binary mode is required for exporters (:issue:`2564`, fixes :issue:`2553`) +- Mention issue with :meth:`FormRequest.from_response + ` due to bug in lxml (:issue:`2572`) +- Use single quotes uniformly in templates (:issue:`2596`) +- Document :reqmeta:`ftp_user` and :reqmeta:`ftp_password` meta keys (:issue:`2587`) +- Removed section on deprecated ``contrib/`` (:issue:`2636`) +- Recommend Anaconda when installing Scrapy on Windows + (:issue:`2477`, fixes :issue:`2475`) +- FAQ: rewrite note on Python 3 support on Windows (:issue:`2690`) +- Rearrange selector sections (:issue:`2705`) +- Remove ``__nonzero__`` from :class:`SelectorList` docs (:issue:`2683`) Scrapy 1.3.3 (2017-03-10) From e139d990fca1e1e2144992bb2e05665d17c3f989 Mon Sep 17 00:00:00 2001 From: Paul Tremberth Date: Fri, 21 Apr 2017 16:53:20 +0200 Subject: [PATCH 04/12] Fix sphinx-build warning on deprecated latex_paper_size --- docs/Makefile | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/Makefile b/docs/Makefile index a3d1611f9..187f03c4c 100644 --- a/docs/Makefile +++ b/docs/Makefile @@ -10,7 +10,8 @@ PAPER = SOURCES = SHELL = /bin/bash -ALLSPHINXOPTS = -b $(BUILDER) -d build/doctrees -D latex_paper_size=$(PAPER) \ +ALLSPHINXOPTS = -b $(BUILDER) -d build/doctrees \ + -D latex_elements.papersize=$(PAPER) \ $(SPHINXOPTS) . build/$(BUILDER) $(SOURCES) .PHONY: help update build html htmlhelp clean From c6464cc4f5a152c2d23a79457b7e01ee90b18069 Mon Sep 17 00:00:00 2001 From: Paul Tremberth Date: Tue, 9 May 2017 20:16:41 +0200 Subject: [PATCH 05/12] Add verbose introduction to new features --- docs/news.rst | 46 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+) diff --git a/docs/news.rst b/docs/news.rst index a19d7a93a..f225fef1d 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -6,6 +6,52 @@ Release notes Scrapy 1.4.0 (2017-XX-XX) ------------------------- +Scrapy 1.4 does not bring that many breathtaking new features +but quite a few handy improvements nonetheless. + +Scrapy now supports anonymous FTP sessions with customizable user and +password via the new :setting:`FTP_USER` and :setting:`FTP_PASSWORD` settings. +**And if you're using Twisted version 17.1.0 or above, FTP is now available +with Python 3.** + +Link extractors now work similarly to what a regular modern browser would +do. Especially, leading and trailing whitespace are removed from attributes +(think ``href=" http://example.com"``) when building ``Link`` objects. +This whitespace-stripping also happens for ``action`` attributes with +``FormRequest``. +**Please also note that link extractors do not canonicalize URLs by default +anymore.** This was puzzling users every now and then, and it's not what +browsers do in fact, so we removed that extra transformation on extractred +links. + +There's a new ``response.follow()`` shortcut for creating URLs directly +from the response instance in callbacks. +For example, instead of:: + + scrapy.Request(response.urljoin(somehrefvalue)) + +you can now use the simpler:: + + response.follow(somehrefvalue) + + +For those of you wanting more control on the ``Referer:`` header that Scrapy +sends when following links, you can set your own ``Referrer Policy``. +Prior to Scrapy 1.4, the default ``RefererMiddleware`` would simply and +blindly set it to the URL of the response that generated the HTTP request +(which could leak information on your URL seeds). +By default, Scrapy now behaves much like your regular browser does. +And this policy is fully customizable with W3C standard values +(or with something really custom of your own if you wish). +See :setting:`REFERRER_POLICY` for details. + +Last but not least, Scrapy now has the option to make JSON and XML items +more human-readable, with newlines between items and even custom indenting +offset, using the new :setting:`FEED_EXPORT_INDENT` setting. + +Enjoy! (Or read on for the rest of changes in this release.) + + New Features ~~~~~~~~~~~~ From 7d72394794bea63fdaa08d1264a50b0b83998424 Mon Sep 17 00:00:00 2001 From: Paul Tremberth Date: Wed, 10 May 2017 18:55:09 +0200 Subject: [PATCH 06/12] Reword mention of new response.follow() shortcut --- docs/news.rst | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/docs/news.rst b/docs/news.rst index f225fef1d..27baff754 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -24,8 +24,8 @@ anymore.** This was puzzling users every now and then, and it's not what browsers do in fact, so we removed that extra transformation on extractred links. -There's a new ``response.follow()`` shortcut for creating URLs directly -from the response instance in callbacks. +There's a new ``response.follow()`` shortcut for creating requests directly +from a response instance and a relative URL. For example, instead of:: scrapy.Request(response.urljoin(somehrefvalue)) @@ -34,7 +34,6 @@ you can now use the simpler:: response.follow(somehrefvalue) - For those of you wanting more control on the ``Referer:`` header that Scrapy sends when following links, you can set your own ``Referrer Policy``. Prior to Scrapy 1.4, the default ``RefererMiddleware`` would simply and From 55d10823603e8033a4c4c441b8e78da217a951bf Mon Sep 17 00:00:00 2001 From: Paul Tremberth Date: Thu, 11 May 2017 14:25:11 +0200 Subject: [PATCH 07/12] Reference recent fixes and commits --- docs/news.rst | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/docs/news.rst b/docs/news.rst index 27baff754..64995dd10 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -109,6 +109,8 @@ Bug fixes fixes :issue:`1612`) - 'make htmlview' fix for macOS (:issue:`2661`) - Remove "commands" from the command list (:issue:`2695`) +- Fix duplicate Content-Length header for POST requests with empty body (:issue:`2677`) +- Properly cancel large downloads, i.e. above :setting:`DOWNLOAD_MAXSIZE` (:issue:`1616`) Cleanups & Refactoring ~~~~~~~~~~~~~~~~~~~~~~ @@ -116,7 +118,6 @@ Cleanups & Refactoring - Tests: remove temp files and folders (:issue:`2570`), fixed ProjectUtilsTest on OS X (:issue:`2569`), use portable pypy for Linux on Travis CI (:issue:`2710`) - - Separate building request from ``_requests_to_follow`` in CrawlSpider (:issue:`2562`) - Remove “Python 3 progress” badge (:issue:`2567`) - Add a couple more lines to ``.gitignore`` (:issue:`2557`) @@ -127,6 +128,7 @@ Cleanups & Refactoring - Add omitted ``self`` arguments in default project middleware template (:issue:`2595`) - Remove redundant ``slot.add_request()`` call in ExecutionEngine (:issue:`2617`) - Catch more specific ``os.error`` exception in :class:`FSFilesStore` (:issue:`2644`) +- Change "localhost" test server certificate (:issue:`2720`) Documentation ~~~~~~~~~~~~~ @@ -142,6 +144,10 @@ Documentation - FAQ: rewrite note on Python 3 support on Windows (:issue:`2690`) - Rearrange selector sections (:issue:`2705`) - Remove ``__nonzero__`` from :class:`SelectorList` docs (:issue:`2683`) +- Mention how to disable request filtering in documentation of + :setting:`DUPEFILTER_CLASS` setting (:issue:`2714`) +- Add sphinx_rtd_theme to docs setup readme (:issue:`2668`) +- Open file in text mode in JSON item writer example (:issue:`2729`) Scrapy 1.3.3 (2017-03-10) From 896c30a8eb1ea5d03898b27f238243ea629a3e42 Mon Sep 17 00:00:00 2001 From: Paul Tremberth Date: Fri, 12 May 2017 19:39:31 +0200 Subject: [PATCH 08/12] Reference items pretty-printing issue number --- docs/news.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/news.rst b/docs/news.rst index 64995dd10..c249c038a 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -86,6 +86,8 @@ New Features using the new :setting:`MEDIA_ALLOW_REDIRECTS` setting (:issue:`2616`, fixes :issue:`2004`) - Accept non-complete responses from websites using a new :setting:`DOWNLOAD_FAIL_ON_DATALOSS` setting (:issue:`2590`, fixes :issue:`2586`) +- Optional pretty-printing of JSON and XML items via + :setting:`FEED_EXPORT_INDENT` setting (:issue:`2456`, fixes :issue:`1327`) .. _brotli: https://github.com/google/brotli .. _brotlipy: https://github.com/python-hyper/brotlipy/ From 432668acf7b8236229fca3b7726139acdc0c05f2 Mon Sep 17 00:00:00 2001 From: Paul Tremberth Date: Tue, 16 May 2017 16:00:26 +0200 Subject: [PATCH 09/12] Mention implementation of #667 --- docs/news.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/news.rst b/docs/news.rst index c249c038a..2bd6640b3 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -88,6 +88,8 @@ New Features :setting:`DOWNLOAD_FAIL_ON_DATALOSS` setting (:issue:`2590`, fixes :issue:`2586`) - Optional pretty-printing of JSON and XML items via :setting:`FEED_EXPORT_INDENT` setting (:issue:`2456`, fixes :issue:`1327`) +- Allow dropping fields in ``FormRequest.from_response`` formdata when + ``None`` value is passed (:issue:`667`) .. _brotli: https://github.com/google/brotli .. _brotlipy: https://github.com/python-hyper/brotlipy/ From a3d3cd4cb7d256a0b876019279ec88c47d9957ff Mon Sep 17 00:00:00 2001 From: Paul Tremberth Date: Wed, 17 May 2017 19:52:18 +0200 Subject: [PATCH 10/12] Update with latest merges --- docs/news.rst | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/news.rst b/docs/news.rst index 2bd6640b3..7debe176b 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -90,6 +90,8 @@ New Features :setting:`FEED_EXPORT_INDENT` setting (:issue:`2456`, fixes :issue:`1327`) - Allow dropping fields in ``FormRequest.from_response`` formdata when ``None`` value is passed (:issue:`667`) +- Per-request retry times with the new :reqmeta:`max_retry_times` meta key + (:issue:`2642`) .. _brotli: https://github.com/google/brotli .. _brotlipy: https://github.com/python-hyper/brotlipy/ @@ -133,6 +135,7 @@ Cleanups & Refactoring - Remove redundant ``slot.add_request()`` call in ExecutionEngine (:issue:`2617`) - Catch more specific ``os.error`` exception in :class:`FSFilesStore` (:issue:`2644`) - Change "localhost" test server certificate (:issue:`2720`) +- Remove unused ``MEMUSAGE_REPORT`` setting (:issue:`2576`) Documentation ~~~~~~~~~~~~~ From edcde7a2cf1b7abbc0b27d51d83f40fe6fe8f5fa Mon Sep 17 00:00:00 2001 From: Mikhail Korobov Date: Thu, 18 May 2017 23:11:34 +0500 Subject: [PATCH 11/12] DOC tweak release notes: promote response.follow, mention logging/stats changes --- docs/news.rst | 53 ++++++++++++++++++++++++++++++++++----------------- 1 file changed, 35 insertions(+), 18 deletions(-) diff --git a/docs/news.rst b/docs/news.rst index 7debe176b..b5e31e444 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -11,29 +11,42 @@ but quite a few handy improvements nonetheless. Scrapy now supports anonymous FTP sessions with customizable user and password via the new :setting:`FTP_USER` and :setting:`FTP_PASSWORD` settings. -**And if you're using Twisted version 17.1.0 or above, FTP is now available -with Python 3.** +And if you're using Twisted version 17.1.0 or above, FTP is now available +with Python 3. + +There's a new :meth:`response.follow ` method +for creating requests; **it is now a recommended way to create Requests +in Scrapy spiders**. This method makes it easier to write correct +spiders; ``response.follow`` has several advantages over creating +``scrapy.Request`` objects directly: + +* it handles relative URLs; +* it works properly with non-ascii URLs on non-UTF8 pages; +* in addition to absolute and relative URLs it supports Selectors; + for ```` elements it can also extract their href values. + +For example, instead of this:: + + for href in response.css('li.page a::attr(href)').extract(): + url = response.urljoin(href) + yield scrapy.Request(url, self.parse, encoding=response.encoding) + +One can now write this:: + + for a in response.css('li.page a'): + yield response.follow(a, self.parse) + +Link extractors are also improved. They work similarly to what a regular +modern browser would do: leading and trailing whitespace are removed +from attributes (think ``href=" http://example.com"``) when building +``Link`` objects. This whitespace-stripping also happens for ``action`` +attributes with ``FormRequest``. -Link extractors now work similarly to what a regular modern browser would -do. Especially, leading and trailing whitespace are removed from attributes -(think ``href=" http://example.com"``) when building ``Link`` objects. -This whitespace-stripping also happens for ``action`` attributes with -``FormRequest``. **Please also note that link extractors do not canonicalize URLs by default anymore.** This was puzzling users every now and then, and it's not what browsers do in fact, so we removed that extra transformation on extractred links. -There's a new ``response.follow()`` shortcut for creating requests directly -from a response instance and a relative URL. -For example, instead of:: - - scrapy.Request(response.urljoin(somehrefvalue)) - -you can now use the simpler:: - - response.follow(somehrefvalue) - For those of you wanting more control on the ``Referer:`` header that Scrapy sends when following links, you can set your own ``Referrer Policy``. Prior to Scrapy 1.4, the default ``RefererMiddleware`` would simply and @@ -44,6 +57,10 @@ And this policy is fully customizable with W3C standard values (or with something really custom of your own if you wish). See :setting:`REFERRER_POLICY` for details. +To make Scrapy spiders easier to debug, Scrapy logs more stats by default +in 1.4: memory usage stats, detailed retry stats, detailed HTTP error code +stats. A similar change is that HTTP cache path is also visible in logs now. + Last but not least, Scrapy now has the option to make JSON and XML items more human-readable, with newlines between items and even custom indenting offset, using the new :setting:`FEED_EXPORT_INDENT` setting. @@ -60,7 +77,7 @@ New Features - Enable memusage extension by default (:issue:`2187`) ; **this is technically backwards-incompatible** so please check if you have any non-default ``MEMUSAGE_***`` settings set. -- New :meth:`Response.follow ` shortcut +- New :ref:`response.follow ` shortcut for creating requests (:issue:`1940`) - Added ``flags`` argument and attribute to :class:`Request ` objects (:issue:`2047`) From 76e5b0f65c256373411c8fa6e005a85c48cad6ee Mon Sep 17 00:00:00 2001 From: Mikhail Korobov Date: Fri, 19 May 2017 01:13:32 +0500 Subject: [PATCH 12/12] DOC 1.4 deprecations and backwards incompatible changes, add recent commits to news. --- docs/news.rst | 28 +++++++++++++++++++++------- 1 file changed, 21 insertions(+), 7 deletions(-) diff --git a/docs/news.rst b/docs/news.rst index b5e31e444..3c641af44 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -67,6 +67,21 @@ offset, using the new :setting:`FEED_EXPORT_INDENT` setting. Enjoy! (Or read on for the rest of changes in this release.) +Deprecations and Backwards Incompatible Changes +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +- Default to ``canonicalize=False`` in :class:`scrapy.linkextractors.LinkExtractor` + (:issue:`2537`, fixes :issue:`1941` and :issue:`1982`): + **warning, this is technically backwards-incompatible** +- Enable memusage extension by default (:issue:`2539`, fixes :issue:`2187`); + **this is technically backwards-incompatible** so please check if you have + any non-default ``MEMUSAGE_***`` options set. +- ``EDITOR`` environment variable now takes precedence over ``EDITOR`` + option defined in settings.py (:issue:`1829`); Scrapy default settings + no longer depend on environment variables. **This is technically a backwards + incompatible change**. +- ``Spider.make_requests_from_url`` is deprecated + (:issue:`1728`, fixes :issue:`1495`). New Features ~~~~~~~~~~~~ @@ -74,9 +89,6 @@ New Features - Accept proxy credentials in :reqmeta:`proxy` request meta key (:issue:`2526`) - Support `brotli`_-compressed content; requires optional `brotlipy`_ (:issue:`2535`) -- Enable memusage extension by default (:issue:`2187`) ; - **this is technically backwards-incompatible** so please check if you have - any non-default ``MEMUSAGE_***`` settings set. - New :ref:`response.follow ` shortcut for creating requests (:issue:`1940`) - Added ``flags`` argument and attribute to :class:`Request ` @@ -88,9 +100,6 @@ New Features - Added ``httperror/response_ignored_count`` and ``httperror/response_ignored_status_count/`` stats to :class:`HttpErrorMiddleware ` (:issue:`2566`) -- Default to ``canonicalize=False`` in :class:`scrapy.linkextractors.LinkExtractor` - (:issue:`2537`, fixes :issue:`1941` and :issue:`1982`): - **warning, this is technically backwards-incompatible** - Customizable :setting:`Referrer policy ` in :class:`RefererMiddleware ` (:issue:`2306`) @@ -109,6 +118,8 @@ New Features ``None`` value is passed (:issue:`667`) - Per-request retry times with the new :reqmeta:`max_retry_times` meta key (:issue:`2642`) +- ``python -m scrapy`` as a more explicit alternative to ``scrapy`` command + (:issue:`2740`) .. _brotli: https://github.com/google/brotli .. _brotlipy: https://github.com/python-hyper/brotlipy/ @@ -134,6 +145,8 @@ Bug fixes - Remove "commands" from the command list (:issue:`2695`) - Fix duplicate Content-Length header for POST requests with empty body (:issue:`2677`) - Properly cancel large downloads, i.e. above :setting:`DOWNLOAD_MAXSIZE` (:issue:`1616`) +- ImagesPipeline: fixed processing of transparent PNG images with palette + (:issue:`2675`) Cleanups & Refactoring ~~~~~~~~~~~~~~~~~~~~~~ @@ -144,8 +157,8 @@ Cleanups & Refactoring - Separate building request from ``_requests_to_follow`` in CrawlSpider (:issue:`2562`) - Remove “Python 3 progress” badge (:issue:`2567`) - Add a couple more lines to ``.gitignore`` (:issue:`2557`) -- Deprecate ``Spider.make_requests_from_url`` (:issue:`1728`) - Remove bumpversion prerelease configuration (:issue:`2159`) +- Add codecov.yml file (:issue:`2750`) - Set context factory implementation based on Twisted version (:issue:`2577`, fixes :issue:`2560`) - Add omitted ``self`` arguments in default project middleware template (:issue:`2595`) @@ -172,6 +185,7 @@ Documentation :setting:`DUPEFILTER_CLASS` setting (:issue:`2714`) - Add sphinx_rtd_theme to docs setup readme (:issue:`2668`) - Open file in text mode in JSON item writer example (:issue:`2729`) +- Clarify ``allowed_domains`` example (:issue:`2670`) Scrapy 1.3.3 (2017-03-10)