From 706910790b6ee755bafa828606e215e668af3eee Mon Sep 17 00:00:00 2001 From: Mikhail Korobov Date: Wed, 26 Dec 2018 18:28:24 +0500 Subject: [PATCH 1/7] [wip] draft 1.6 release notes --- docs/news.rst | 153 +++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 150 insertions(+), 3 deletions(-) diff --git a/docs/news.rst b/docs/news.rst index adf679ded..99a339cea 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -6,11 +6,156 @@ Release notes Scrapy 1.6.0 (unreleased) ------------------------- +Highlights for this release: + +* better Windows compatibility; +* Python 3.7 compatibility; +* big documentation improvements, including a switch + from ``.extract() / .extract_first()`` API to ``.get() / .getall()`` API; +* Feed exports, FilePipeline and MediaPipeline improvements; +* ``scrapy.contracts`` fixes and new features; +* large clean-up of deprecated code +* TODO + +parsel 1.5 +~~~~~~~~~~ + +TODO +While this is not a change in Scrapy itself, a new version of ``parsel`` +is released; Scrapy now depends on ``parsel >= 1.5``. + +Feed export improvements +~~~~~~~~~~~~~~~~~~~~~~~~ + +* ``from_crawler`` support is added to feed exporters and feed storages. This, + among other things, allow to access Scrapy settings from custom storages + and exporters (:issue:`1605`, :issue:`3348`). +* fixed issue with extra blank lines in .csv exports under Windows + (:issue:`3039`); +* better error message when an exporter is disabled (:issue:`3358`); + +FilePipeline and MediaPipeline improvements +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +* Expose more options for S3FilesStore: :setting:`AWS_ENDPOINT_URL`, + :setting:`AWS_USE_SSL`, :setting:`AWS_VERIFY`, :setting:`AWS_REGION_NAME`. + For example, this allows to use alternative or self-hosted + AWS-compatible providers (:issue:`2609`). +* ACL support for Google Cloud Storage: :setting:`FILES_STORE_GCS_ACL` and + :setting:`IMAGES_STORE_GCS_ACL` (:issue:`3199`). + +``scrapy.contracts`` improvements +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +* Exceptions in contracts code are handled better (:issue:`3377`); +* ``dont_filter=True`` is used for contract requests, which allows to test + different callbacks with the same URL (:issue:`3381`); +* ``request_cls`` attribute in Contract subclasses allow to use different + Request classes in contracts, for example FormRequest (:issue:`3383`). +* Fixed errback handling in contracts, e.g. for cases where a contract + is executed for URL which returns non-200 response (:issue:`3371`). + +Documentation improvements +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +* Docs are re-written to suggest .get/.getall API instead of + .extract/.extract_first. Also, :ref:`topics-selectors` docs are updated + and re-structured to match latest parsel docs; they now contain more topics, + such as :ref:`selecting-attributes` or :ref:`topics-selectors-css-extensions` + (:issue:`3390`). +* :ref:`topics-developer-tools` is a new tutorial which replaces + old Firefox and Firebug tutorials (:issue:`3400`). +* SCRAPY_PROJECT environment variable is documented (:issue:`3518`); +* troubleshooting section is added to install instructions (:issue:`3517`); +* improved links to beginner resources in the tutorial + (:issue:`3367`, :issue:`3468`); +* fixed :setting:`RETRY_HTTP_CODES` default values in docs (:issue:`3335`); +* remove unused `DEPTH_STATS` option from docs (:issue:`3245`); +* other cleanups (:issue:`3347`, :issue:`3350`, :issue:`3445`). + +Better Windows support +~~~~~~~~~~~~~~~~~~~~~~ + +* All Scrapy tests now pass on Windows; Scrapy testing suite is executed + in a Windows environment on CI (:issue:`3315`). +* Scrapy used to produce unnecessary blank lines in .csv exports on Windows, + this is fixed (:issue:`3039`). + +Testing fixes +~~~~~~~~~~~~~ + +* Python 3.7 support (:issue:`3326`, :issue:`3150`, :issue:`3547`) +* Testing and CI fixes (:issue:`3526`, :issue:`3538`, :issue:`3308`, + :issue:`3311`, :issue:`3309`, :issue:`3305`, :issue:`3210`, :issue:`3299`) + +Deprecation removals +~~~~~~~~~~~~~~~~~~~~ + +Compatibility shims for pre-1.0 Scrapy module names are removed +(:issue:`3318`): + +* ``scrapy.command`` +* ``scrapy.contrib`` (with all submodules) +* ``scrapy.contrib_exp`` (with all submodules) +* ``scrapy.dupefilter`` +* ``scrapy.linkextractor`` +* ``scrapy.project`` +* ``scrapy.spider`` +* ``scrapy.spidermanager`` +* ``scrapy.squeue`` +* ``scrapy.stats`` +* ``scrapy.statscol`` +* ``scrapy.utils.decorator`` + +See :ref:`module_relocations` for more information, or use suggestions +from Scrapy 1.5.x deprecation warnings to update your code. + +Other deprecation removals: + +* Deprecated scrapy.interfaces.ISpiderManager is removed; please use + scrapy.interfaces.ISpiderLoader. +* Deprecated ``CrawlerSettings`` class is removed (:issue:`3327`). +* Deprecated ``Settings.overrides`` and ``Settings.defaults`` attributes + are removed (:issue:`3327`, :issue:`3359`). + +Internal improvements +~~~~~~~~~~~~~~~~~~~~~ + +* ``from_crawler`` support is added to dupefilters (:issue:`2956`); this allows + to access e.g. settings or a spider from a dupefilter. +* :signal:`item_error` is fired when an error happens in a pipeline + (:issue:`3256`); +* :signal:`request_reached_downloader` is fired when Downloader gets + a new Request; this signal can be useful e.g. for custom Schedulers + (:issue:`3393`). +* ``scrapy.http.cookies.CookieJar.clear`` accepts "domain", "path" and "name" + optional arguments (:issue:`3231`). + +Usability improvements +~~~~~~~~~~~~~~~~~~~~~~ + +* more stats for RobotsTxtMiddleware (:issue:`3100`) +* INFO log level is used to show telnet host/port (:issue:`3115`) +* a message is added to IgnoreRequest in RobotsTxtMiddleware (:issue:`3113`) +* better validation of ``url`` argument in ``Response.follow`` (:issue:`3131`) +* non-zero exit code is returned from Scrapy commands when error happens + on spider inititalization (:issue:`3226`) +* Link extraction improvements: "ftp" is added to scheme list (:issue:`3152`); + "flv" is added to common video extensions (:issue:`3165`) + +Bug fixes +~~~~~~~~~ +* proper handling of pickling errors in Python 3 when serializing objects + for disk queues (:issue:`3082`) +* flags are now preserved when copying Requests (:issue:`3342`); +* FormRequest.from_response clickdata shouldn't ignore elements with + ``input[type=image]`` (:issue:`3153`). +* FormRequest.from_response should preserve duplicate keys (:issue:`3247`) + Cleanups ~~~~~~~~ - -* Remove deprecated ``CrawlerSettings`` class. -* Remove deprecated ``Settings.overrides`` and ``Settings.defaults`` attributes. +* additional files are included to sdist (:issue:`3495`); +* code style fixes (:issue:`3405`, :issue:`3304`) Scrapy 1.5.2 (2019-01-22) ------------------------- @@ -1080,6 +1225,8 @@ until it reaches a stable status. See more examples for scripts running Scrapy: :ref:`topics-practices` +.. _module_relocations: + Module Relocations ~~~~~~~~~~~~~~~~~~ From e479f5aa15809e7f75a7dbc20d0629f57be46b5d Mon Sep 17 00:00:00 2001 From: Mikhail Korobov Date: Thu, 27 Dec 2018 00:48:10 +0500 Subject: [PATCH 2/7] DOC update changelog * changes from recently merged pull requests * more highlights * re-organized headers * Selector API changes --- docs/news.rst | 142 +++++++++++++++++++++++++++++++++++++------------- 1 file changed, 105 insertions(+), 37 deletions(-) diff --git a/docs/news.rst b/docs/news.rst index 99a339cea..bf469a350 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -6,41 +6,83 @@ Release notes Scrapy 1.6.0 (unreleased) ------------------------- -Highlights for this release: +Highlights: -* better Windows compatibility; +* better Windows support; * Python 3.7 compatibility; * big documentation improvements, including a switch - from ``.extract() / .extract_first()`` API to ``.get() / .getall()`` API; -* Feed exports, FilePipeline and MediaPipeline improvements; + from ``.extract()`` + ``.extract_first()`` API to ``.get()`` + ``.getall()`` + API; +* feed exports, FilePipeline and MediaPipeline improvements; +* better extensibility: :signal:`item_error` and + :signal:`request_reached_downloader` signals; ``from_crawler`` support + for feed exporters, feed storages and dupefilters. * ``scrapy.contracts`` fixes and new features; -* large clean-up of deprecated code -* TODO +* telnet console security improvements; +* clean-up of the deprecated code; +* various bug fixes, small new features and usability improvements across + the codebase. -parsel 1.5 -~~~~~~~~~~ +Selector API changes +~~~~~~~~~~~~~~~~~~~~ -TODO -While this is not a change in Scrapy itself, a new version of ``parsel`` -is released; Scrapy now depends on ``parsel >= 1.5``. +While these are not changes in Scrapy itself, but rather in the parsel_ +library which Scrapy uses for xpath/css selectors, these changes are +worth mentioning here. Scrapy now depends on parsel >= 1.5, and +Scrapy documentation is updated to follow recent ``parsel`` API conventions. -Feed export improvements -~~~~~~~~~~~~~~~~~~~~~~~~ +Most visible change is that ``.get()`` and ``.getall()`` selector +methods are now preferred over ``.extract()`` and ``.extract_first()``. +We feel that these new methods result in a more concise and readable code. +See :ref:`old-extraction-api` for more details. + +.. note:: + There are currently **no plans** to deprecate ``.extract()`` + and ``.extract_first()`` methods. + +Another useful new feature is the introduction of ``Selector.attrib`` and +``SelectorList.attrib`` properties, which make it easier to get +attributes of HTML elements. See :ref:`selecting-attributes`. + +CSS selectors are cached in parsel >= 1.5, which makes them faster +when the same CSS path is used many times. This is very common in +case of Scrapy spiders: callbacks are usually called several times, +on different pages. + +If you're using custom ``Selector`` or ``SelectorList`` subclasses, +a **backwards incompatible** change in parsel may affect your code. +See `parsel changelog`_ for a detailed description, as well as for the +full list of improvements. + +.. _parsel changelog: https://parsel.readthedocs.io/en/latest/history.html + +Telnet console +~~~~~~~~~~~~~~ + +**Backwards incompatible**: Scrapy's telnet console now requires username +and password. See :ref:`topics-telnetconsole` for more details. + +New extensibility features +~~~~~~~~~~~~~~~~~~~~~~~~~~ * ``from_crawler`` support is added to feed exporters and feed storages. This, - among other things, allow to access Scrapy settings from custom storages - and exporters (:issue:`1605`, :issue:`3348`). -* fixed issue with extra blank lines in .csv exports under Windows - (:issue:`3039`); -* better error message when an exporter is disabled (:issue:`3358`); + among other things, allows to access Scrapy settings from custom feed + storages and exporters (:issue:`1605`, :issue:`3348`). +* ``from_crawler`` support is added to dupefilters (:issue:`2956`); this allows + to access e.g. settings or a spider from a dupefilter. +* :signal:`item_error` is fired when an error happens in a pipeline + (:issue:`3256`); +* :signal:`request_reached_downloader` is fired when Downloader gets + a new Request; this signal can be useful e.g. for custom Schedulers + (:issue:`3393`). -FilePipeline and MediaPipeline improvements +New FilePipeline and MediaPipeline features ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ * Expose more options for S3FilesStore: :setting:`AWS_ENDPOINT_URL`, :setting:`AWS_USE_SSL`, :setting:`AWS_VERIFY`, :setting:`AWS_REGION_NAME`. For example, this allows to use alternative or self-hosted - AWS-compatible providers (:issue:`2609`). + AWS-compatible providers (:issue:`2609`, :issue:`3548`). * ACL support for Google Cloud Storage: :setting:`FILES_STORE_GCS_ACL` and :setting:`IMAGES_STORE_GCS_ACL` (:issue:`3199`). @@ -55,6 +97,47 @@ FilePipeline and MediaPipeline improvements * Fixed errback handling in contracts, e.g. for cases where a contract is executed for URL which returns non-200 response (:issue:`3371`). +Usability and other improvements, cleanups +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +* All Scrapy tests now pass on Windows; Scrapy testing suite is executed + in a Windows environment on CI (:issue:`3315`). +* Python 3.7 support (:issue:`3326`, :issue:`3150`, :issue:`3547`). +* Lazy loading of Downloader Handlers is now optional; this enables better + initialization error handling in custom Downloader Handlers (:issue:`3394`). +* Testing and CI fixes (:issue:`3526`, :issue:`3538`, :issue:`3308`, + :issue:`3311`, :issue:`3309`, :issue:`3305`, :issue:`3210`, :issue:`3299`) +* better error message when an exporter is disabled (:issue:`3358`); +* ``scrapy.http.cookies.CookieJar.clear`` accepts "domain", "path" and "name" + optional arguments (:issue:`3231`). +* more stats for RobotsTxtMiddleware (:issue:`3100`) +* INFO log level is used to show telnet host/port (:issue:`3115`) +* a message is added to IgnoreRequest in RobotsTxtMiddleware (:issue:`3113`) +* better validation of ``url`` argument in ``Response.follow`` (:issue:`3131`) +* non-zero exit code is returned from Scrapy commands when error happens + on spider inititalization (:issue:`3226`); +* link extraction improvements: "ftp" is added to scheme list (:issue:`3152`); + "flv" is added to common video extensions (:issue:`3165`) +* `scrapy shell --help` mentions syntax required for local files + (``./file.html``) - :issue:`3496`. +* additional files are included to sdist (:issue:`3495`); +* code style fixes (:issue:`3405`, :issue:`3304`); +* unneeded .strip() call is removed (:issue:`3519`); +* collections.deque is used to store MiddlewareManager methods instead + of a list (:issue:`3476`) + +Bug fixes +~~~~~~~~~ + +* fixed issue with extra blank lines in .csv exports under Windows + (:issue:`3039`); +* proper handling of pickling errors in Python 3 when serializing objects + for disk queues (:issue:`3082`) +* flags are now preserved when copying Requests (:issue:`3342`); +* FormRequest.from_response clickdata shouldn't ignore elements with + ``input[type=image]`` (:issue:`3153`). +* FormRequest.from_response should preserve duplicate keys (:issue:`3247`) + Documentation improvements ~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -73,21 +156,6 @@ Documentation improvements * remove unused `DEPTH_STATS` option from docs (:issue:`3245`); * other cleanups (:issue:`3347`, :issue:`3350`, :issue:`3445`). -Better Windows support -~~~~~~~~~~~~~~~~~~~~~~ - -* All Scrapy tests now pass on Windows; Scrapy testing suite is executed - in a Windows environment on CI (:issue:`3315`). -* Scrapy used to produce unnecessary blank lines in .csv exports on Windows, - this is fixed (:issue:`3039`). - -Testing fixes -~~~~~~~~~~~~~ - -* Python 3.7 support (:issue:`3326`, :issue:`3150`, :issue:`3547`) -* Testing and CI fixes (:issue:`3526`, :issue:`3538`, :issue:`3308`, - :issue:`3311`, :issue:`3309`, :issue:`3305`, :issue:`3210`, :issue:`3299`) - Deprecation removals ~~~~~~~~~~~~~~~~~~~~ @@ -107,7 +175,7 @@ Compatibility shims for pre-1.0 Scrapy module names are removed * ``scrapy.statscol`` * ``scrapy.utils.decorator`` -See :ref:`module_relocations` for more information, or use suggestions +See :ref:`module-relocations` for more information, or use suggestions from Scrapy 1.5.x deprecation warnings to update your code. Other deprecation removals: @@ -1225,7 +1293,7 @@ until it reaches a stable status. See more examples for scripts running Scrapy: :ref:`topics-practices` -.. _module_relocations: +.. _module-relocations: Module Relocations ~~~~~~~~~~~~~~~~~~ From 638469f9efdcc104f7b1a1c1a9890694e0d41c68 Mon Sep 17 00:00:00 2001 From: Mikhail Korobov Date: Fri, 28 Dec 2018 01:13:01 +0500 Subject: [PATCH 3/7] DOC extract_first/extract matches get/getall better Thanks @Gallaecio! --- docs/news.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/news.rst b/docs/news.rst index bf469a350..4a236f1b9 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -11,7 +11,7 @@ Highlights: * better Windows support; * Python 3.7 compatibility; * big documentation improvements, including a switch - from ``.extract()`` + ``.extract_first()`` API to ``.get()`` + ``.getall()`` + from ``.extract_first()`` + ``.extract()`` API to ``.get()`` + ``.getall()`` API; * feed exports, FilePipeline and MediaPipeline improvements; * better extensibility: :signal:`item_error` and @@ -32,7 +32,7 @@ worth mentioning here. Scrapy now depends on parsel >= 1.5, and Scrapy documentation is updated to follow recent ``parsel`` API conventions. Most visible change is that ``.get()`` and ``.getall()`` selector -methods are now preferred over ``.extract()`` and ``.extract_first()``. +methods are now preferred over ``.extract_first()`` and ``.extract()``. We feel that these new methods result in a more concise and readable code. See :ref:`old-extraction-api` for more details. From 4cf4dd1d3e068e0df32f700c89d833cc7cd79b85 Mon Sep 17 00:00:00 2001 From: Mikhail Korobov Date: Wed, 30 Jan 2019 03:08:17 +0500 Subject: [PATCH 4/7] DOC add recent changes to changelog --- docs/news.rst | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/docs/news.rst b/docs/news.rst index 4a236f1b9..1a08f93ec 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -75,6 +75,9 @@ New extensibility features * :signal:`request_reached_downloader` is fired when Downloader gets a new Request; this signal can be useful e.g. for custom Schedulers (:issue:`3393`). +* new SitemapSpider :meth:`~.SitemapSpider.sitemap_filter` method which allows + to select sitemap entries based on their attributes in SitemapSpider + subclasses (:issue:`3512`). New FilePipeline and MediaPipeline features ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -125,6 +128,7 @@ Usability and other improvements, cleanups * unneeded .strip() call is removed (:issue:`3519`); * collections.deque is used to store MiddlewareManager methods instead of a list (:issue:`3476`) +* Referer header value is added to RFPDupeFilter log messages (:issue:`3588`) Bug fixes ~~~~~~~~~ @@ -154,7 +158,8 @@ Documentation improvements (:issue:`3367`, :issue:`3468`); * fixed :setting:`RETRY_HTTP_CODES` default values in docs (:issue:`3335`); * remove unused `DEPTH_STATS` option from docs (:issue:`3245`); -* other cleanups (:issue:`3347`, :issue:`3350`, :issue:`3445`). +* other cleanups (:issue:`3347`, :issue:`3350`, :issue:`3445`, :issue:`3544`, + :issue:`3605`). Deprecation removals ~~~~~~~~~~~~~~~~~~~~ From 0fc9d705c271f5d87174143c09f95993e5a45797 Mon Sep 17 00:00:00 2001 From: Mikhail Korobov Date: Wed, 30 Jan 2019 03:28:19 +0500 Subject: [PATCH 5/7] DOC mention that telnet security improvements happened in 1.5.2 --- docs/news.rst | 45 ++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 44 insertions(+), 1 deletion(-) diff --git a/docs/news.rst b/docs/news.rst index 1a08f93ec..a4f07efad 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -3,6 +3,8 @@ Release notes ============= +.. _release-1.6.0: + Scrapy 1.6.0 (unreleased) ------------------------- @@ -18,11 +20,13 @@ Highlights: :signal:`request_reached_downloader` signals; ``from_crawler`` support for feed exporters, feed storages and dupefilters. * ``scrapy.contracts`` fixes and new features; -* telnet console security improvements; +* telnet console security improvements, first released as a + backport in :ref:`release-1.5.2`; * clean-up of the deprecated code; * various bug fixes, small new features and usability improvements across the codebase. + Selector API changes ~~~~~~~~~~~~~~~~~~~~ @@ -230,6 +234,8 @@ Cleanups * additional files are included to sdist (:issue:`3495`); * code style fixes (:issue:`3405`, :issue:`3304`) +.. _release-1.5.2: + Scrapy 1.5.2 (2019-01-22) ------------------------- @@ -247,6 +253,8 @@ Scrapy 1.5.2 (2019-01-22) * Backport CI build failure under GCE environemnt due to boto import error. +.. _release-1.5.1: + Scrapy 1.5.1 (2018-07-12) ------------------------- @@ -262,6 +270,9 @@ This is a maintenance release with important bug fixes, but no new features: :issue:`3279`, :issue:`3201`, :issue:`3260`, :issue:`3284`, :issue:`3298`, :issue:`3294`). + +.. _release-1.5.0: + Scrapy 1.5.0 (2017-12-29) ------------------------- @@ -373,6 +384,7 @@ Docs - Document ``from_crawler`` methods for spider and downloader middlewares (:issue:`3019`) +.. _release-1.4.0: Scrapy 1.4.0 (2017-05-18) ------------------------- @@ -559,6 +571,8 @@ Documentation - Clarify ``allowed_domains`` example (:issue:`2670`) +.. _release-1.3.3: + Scrapy 1.3.3 (2017-03-10) ------------------------- @@ -571,6 +585,7 @@ Bug fixes A new setting is introduced to toggle between warning or exception if needed ; see :setting:`SPIDER_LOADER_WARN_ONLY` for details. +.. _release-1.3.2: Scrapy 1.3.2 (2017-02-13) ------------------------- @@ -582,6 +597,8 @@ Bug fixes - Use consistent selectors for author field in tutorial (:issue:`2551`). - Fix TLS compatibility in Twisted 17+ (:issue:`2558`) +.. _release-1.3.1: + Scrapy 1.3.1 (2017-02-08) ------------------------- @@ -630,6 +647,8 @@ Cleanups - Remove dead code supporting old Twisted versions (:issue:`2544`). +.. _release-1.3.0: + Scrapy 1.3.0 (2016-12-21) ------------------------- @@ -669,6 +688,7 @@ Dependencies & Cleanups - ``ChunkedTransferMiddleware`` is deprecated and removed from the default downloader middlewares. +.. _release-1.2.3: Scrapy 1.2.3 (2017-03-03) ------------------------- @@ -676,6 +696,8 @@ Scrapy 1.2.3 (2017-03-03) - Packaging fix: disallow unsupported Twisted versions in setup.py +.. _release-1.2.2: + Scrapy 1.2.2 (2016-12-06) ------------------------- @@ -711,6 +733,8 @@ Other changes .. _conda-forge: https://anaconda.org/conda-forge/scrapy +.. _release-1.2.1: + Scrapy 1.2.1 (2016-10-21) ------------------------- @@ -735,6 +759,8 @@ Other changes - Removed ``www.`` from ``start_urls`` in built-in spider templates (:issue:`2299`). +.. _release-1.2.0: + Scrapy 1.2.0 (2016-10-03) ------------------------- @@ -803,12 +829,14 @@ Documentation - Reworded misleading :setting:`RANDOMIZE_DOWNLOAD_DELAY` description (:issue:`2190`). - Add StackOverflow as a support channel (:issue:`2257`). +.. _release-1.1.4: Scrapy 1.1.4 (2017-03-03) ------------------------- - Packaging fix: disallow unsupported Twisted versions in setup.py +.. _release-1.1.3: Scrapy 1.1.3 (2016-09-22) ------------------------- @@ -826,6 +854,7 @@ Documentation rewritten to use http://toscrape.com websites (:issue:`2236`, :issue:`2249`, :issue:`2252`). +.. _release-1.1.2: Scrapy 1.1.2 (2016-08-18) ------------------------- @@ -840,6 +869,7 @@ Bug fixes - :setting:`IMAGES_EXPIRES` default value set back to 90 (the regression was introduced in 1.1.1) +.. _release-1.1.1: Scrapy 1.1.1 (2016-07-13) ------------------------- @@ -892,6 +922,7 @@ Tests - Upgrade py.test requirement on Travis CI and Pin pytest-cov to 2.2.1 (:issue:`2095`) +.. _release-1.1.0: Scrapy 1.1.0 (2016-05-11) ------------------------- @@ -1081,12 +1112,14 @@ Bugfixes - HTTPS+CONNECT tunnels could get mixed up when using multiple proxies to same remote host (:issue:`1912`). +.. _release-1.0.7: Scrapy 1.0.7 (2017-03-03) ------------------------- - Packaging fix: disallow unsupported Twisted versions in setup.py +.. _release-1.0.6: Scrapy 1.0.6 (2016-05-04) ------------------------- @@ -1096,6 +1129,7 @@ Scrapy 1.0.6 (2016-05-04) - DOC: Support for Sphinx 1.4+ (:issue:`1893`) - DOC: Consistency in selectors examples (:issue:`1869`) +.. _release-1.0.5: Scrapy 1.0.5 (2016-02-04) ------------------------- @@ -1105,6 +1139,7 @@ Scrapy 1.0.5 (2016-02-04) - DOC: Fixed typos in tutorial and media-pipeline (:commit:`808a9ea` and :commit:`803bd87`) - DOC: Add AjaxCrawlMiddleware to DOWNLOADER_MIDDLEWARES_BASE in settings docs (:commit:`aa94121`) +.. _release-1.0.4: Scrapy 1.0.4 (2015-12-30) ------------------------- @@ -1158,12 +1193,16 @@ Scrapy 1.0.4 (2015-12-30) - Small grammatical change (:commit:`8752294`) - Add openssl version to version command (:commit:`13c45ac`) +.. _release-1.0.3: + Scrapy 1.0.3 (2015-08-11) ------------------------- - add service_identity to scrapy install_requires (:commit:`cbc2501`) - Workaround for travis#296 (:commit:`66af9cd`) +.. _release-1.0.2: + Scrapy 1.0.2 (2015-08-06) ------------------------- @@ -1174,6 +1213,8 @@ Scrapy 1.0.2 (2015-08-06) - Fixed typos (:commit:`a9ae7b0`) - Fix doc reference. (:commit:`7c8a4fe`) +.. _release-1.0.1: + Scrapy 1.0.1 (2015-07-01) ------------------------- @@ -1184,6 +1225,8 @@ Scrapy 1.0.1 (2015-07-01) - DOC remove version suffix from ubuntu package (:commit:`5303c66`) - DOC Update release date for 1.0 (:commit:`c89fa29`) +.. _release-1.0.0: + Scrapy 1.0.0 (2015-06-19) ------------------------- From 2c8c8b2dd8683787826713ed1d0fbfb2ec1af04a Mon Sep 17 00:00:00 2001 From: Mikhail Korobov Date: Wed, 30 Jan 2019 17:30:13 +0500 Subject: [PATCH 6/7] DOC fix after bad merge - remove duplicate entries in changelog --- docs/news.rst | 72 +++++++++++++-------------------------------------- 1 file changed, 18 insertions(+), 54 deletions(-) diff --git a/docs/news.rst b/docs/news.rst index a4f07efad..4711d2f35 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -26,7 +26,6 @@ Highlights: * various bug fixes, small new features and usability improvements across the codebase. - Selector API changes ~~~~~~~~~~~~~~~~~~~~ @@ -82,6 +81,8 @@ New extensibility features * new SitemapSpider :meth:`~.SitemapSpider.sitemap_filter` method which allows to select sitemap entries based on their attributes in SitemapSpider subclasses (:issue:`3512`). +* Lazy loading of Downloader Handlers is now optional; this enables better + initialization error handling in custom Downloader Handlers (:issue:`3394`). New FilePipeline and MediaPipeline features ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -104,34 +105,20 @@ New FilePipeline and MediaPipeline features * Fixed errback handling in contracts, e.g. for cases where a contract is executed for URL which returns non-200 response (:issue:`3371`). -Usability and other improvements, cleanups -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Usability improvements +~~~~~~~~~~~~~~~~~~~~~~ -* All Scrapy tests now pass on Windows; Scrapy testing suite is executed - in a Windows environment on CI (:issue:`3315`). -* Python 3.7 support (:issue:`3326`, :issue:`3150`, :issue:`3547`). -* Lazy loading of Downloader Handlers is now optional; this enables better - initialization error handling in custom Downloader Handlers (:issue:`3394`). -* Testing and CI fixes (:issue:`3526`, :issue:`3538`, :issue:`3308`, - :issue:`3311`, :issue:`3309`, :issue:`3305`, :issue:`3210`, :issue:`3299`) -* better error message when an exporter is disabled (:issue:`3358`); -* ``scrapy.http.cookies.CookieJar.clear`` accepts "domain", "path" and "name" - optional arguments (:issue:`3231`). * more stats for RobotsTxtMiddleware (:issue:`3100`) * INFO log level is used to show telnet host/port (:issue:`3115`) * a message is added to IgnoreRequest in RobotsTxtMiddleware (:issue:`3113`) * better validation of ``url`` argument in ``Response.follow`` (:issue:`3131`) * non-zero exit code is returned from Scrapy commands when error happens - on spider inititalization (:issue:`3226`); -* link extraction improvements: "ftp" is added to scheme list (:issue:`3152`); + on spider inititalization (:issue:`3226`) +* Link extraction improvements: "ftp" is added to scheme list (:issue:`3152`); "flv" is added to common video extensions (:issue:`3165`) +* better error message when an exporter is disabled (:issue:`3358`); * `scrapy shell --help` mentions syntax required for local files (``./file.html``) - :issue:`3496`. -* additional files are included to sdist (:issue:`3495`); -* code style fixes (:issue:`3405`, :issue:`3304`); -* unneeded .strip() call is removed (:issue:`3519`); -* collections.deque is used to store MiddlewareManager methods instead - of a list (:issue:`3476`) * Referer header value is added to RFPDupeFilter log messages (:issue:`3588`) Bug fixes @@ -195,44 +182,21 @@ Other deprecation removals: * Deprecated ``Settings.overrides`` and ``Settings.defaults`` attributes are removed (:issue:`3327`, :issue:`3359`). -Internal improvements -~~~~~~~~~~~~~~~~~~~~~ +Other improvements, cleanups +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -* ``from_crawler`` support is added to dupefilters (:issue:`2956`); this allows - to access e.g. settings or a spider from a dupefilter. -* :signal:`item_error` is fired when an error happens in a pipeline - (:issue:`3256`); -* :signal:`request_reached_downloader` is fired when Downloader gets - a new Request; this signal can be useful e.g. for custom Schedulers - (:issue:`3393`). +* All Scrapy tests now pass on Windows; Scrapy testing suite is executed + in a Windows environment on CI (:issue:`3315`). +* Python 3.7 support (:issue:`3326`, :issue:`3150`, :issue:`3547`). +* Testing and CI fixes (:issue:`3526`, :issue:`3538`, :issue:`3308`, + :issue:`3311`, :issue:`3309`, :issue:`3305`, :issue:`3210`, :issue:`3299`) * ``scrapy.http.cookies.CookieJar.clear`` accepts "domain", "path" and "name" optional arguments (:issue:`3231`). - -Usability improvements -~~~~~~~~~~~~~~~~~~~~~~ - -* more stats for RobotsTxtMiddleware (:issue:`3100`) -* INFO log level is used to show telnet host/port (:issue:`3115`) -* a message is added to IgnoreRequest in RobotsTxtMiddleware (:issue:`3113`) -* better validation of ``url`` argument in ``Response.follow`` (:issue:`3131`) -* non-zero exit code is returned from Scrapy commands when error happens - on spider inititalization (:issue:`3226`) -* Link extraction improvements: "ftp" is added to scheme list (:issue:`3152`); - "flv" is added to common video extensions (:issue:`3165`) - -Bug fixes -~~~~~~~~~ -* proper handling of pickling errors in Python 3 when serializing objects - for disk queues (:issue:`3082`) -* flags are now preserved when copying Requests (:issue:`3342`); -* FormRequest.from_response clickdata shouldn't ignore elements with - ``input[type=image]`` (:issue:`3153`). -* FormRequest.from_response should preserve duplicate keys (:issue:`3247`) - -Cleanups -~~~~~~~~ * additional files are included to sdist (:issue:`3495`); -* code style fixes (:issue:`3405`, :issue:`3304`) +* code style fixes (:issue:`3405`, :issue:`3304`); +* unneeded .strip() call is removed (:issue:`3519`); +* collections.deque is used to store MiddlewareManager methods instead + of a list (:issue:`3476`) .. _release-1.5.2: From 91791cd329936ee6ac53523460f9b72c20c66afb Mon Sep 17 00:00:00 2001 From: Mikhail Korobov Date: Wed, 30 Jan 2019 17:53:58 +0500 Subject: [PATCH 7/7] DOC final changelog cleanups --- docs/news.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/news.rst b/docs/news.rst index 4711d2f35..543901809 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -63,7 +63,8 @@ Telnet console ~~~~~~~~~~~~~~ **Backwards incompatible**: Scrapy's telnet console now requires username -and password. See :ref:`topics-telnetconsole` for more details. +and password. See :ref:`topics-telnetconsole` for more details. This change +fixes a **security issue**; see :ref:`release-1.5.2` release notes for details. New extensibility features ~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -117,7 +118,7 @@ Usability improvements * Link extraction improvements: "ftp" is added to scheme list (:issue:`3152`); "flv" is added to common video extensions (:issue:`3165`) * better error message when an exporter is disabled (:issue:`3358`); -* `scrapy shell --help` mentions syntax required for local files +* ``scrapy shell --help`` mentions syntax required for local files (``./file.html``) - :issue:`3496`. * Referer header value is added to RFPDupeFilter log messages (:issue:`3588`)