mirror of https://github.com/scrapy/scrapy.git
Merge pull request #3549 from scrapy/release-notes-1.6
1.6 release notes
This commit is contained in:
Daniel Graña
GitHub
commit
1312174607
236
docs/news.rst
236
docs/news.rst
|
|
@ -3,14 +3,203 @@
|
|||
Release notes
|
||||
=============
|
||||
|
||||
.. _release-1.6.0:
|
||||
|
||||
Scrapy 1.6.0 (unreleased)
|
||||
-------------------------
|
||||
|
||||
Cleanups
|
||||
~~~~~~~~
|
||||
Highlights:
|
||||
|
||||
* Remove deprecated ``CrawlerSettings`` class.
|
||||
* Remove deprecated ``Settings.overrides`` and ``Settings.defaults`` attributes.
|
||||
* better Windows support;
|
||||
* Python 3.7 compatibility;
|
||||
* big documentation improvements, including a switch
|
||||
from ``.extract_first()`` + ``.extract()`` API to ``.get()`` + ``.getall()``
|
||||
API;
|
||||
* feed exports, FilePipeline and MediaPipeline improvements;
|
||||
* better extensibility: :signal:`item_error` and
|
||||
:signal:`request_reached_downloader` signals; ``from_crawler`` support
|
||||
for feed exporters, feed storages and dupefilters.
|
||||
* ``scrapy.contracts`` fixes and new features;
|
||||
* telnet console security improvements, first released as a
|
||||
backport in :ref:`release-1.5.2`;
|
||||
* clean-up of the deprecated code;
|
||||
* various bug fixes, small new features and usability improvements across
|
||||
the codebase.
|
||||
|
||||
Selector API changes
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
While these are not changes in Scrapy itself, but rather in the parsel_
|
||||
library which Scrapy uses for xpath/css selectors, these changes are
|
||||
worth mentioning here. Scrapy now depends on parsel >= 1.5, and
|
||||
Scrapy documentation is updated to follow recent ``parsel`` API conventions.
|
||||
|
||||
Most visible change is that ``.get()`` and ``.getall()`` selector
|
||||
methods are now preferred over ``.extract_first()`` and ``.extract()``.
|
||||
We feel that these new methods result in a more concise and readable code.
|
||||
See :ref:`old-extraction-api` for more details.
|
||||
|
||||
.. note::
|
||||
There are currently **no plans** to deprecate ``.extract()``
|
||||
and ``.extract_first()`` methods.
|
||||
|
||||
Another useful new feature is the introduction of ``Selector.attrib`` and
|
||||
``SelectorList.attrib`` properties, which make it easier to get
|
||||
attributes of HTML elements. See :ref:`selecting-attributes`.
|
||||
|
||||
CSS selectors are cached in parsel >= 1.5, which makes them faster
|
||||
when the same CSS path is used many times. This is very common in
|
||||
case of Scrapy spiders: callbacks are usually called several times,
|
||||
on different pages.
|
||||
|
||||
If you're using custom ``Selector`` or ``SelectorList`` subclasses,
|
||||
a **backwards incompatible** change in parsel may affect your code.
|
||||
See `parsel changelog`_ for a detailed description, as well as for the
|
||||
full list of improvements.
|
||||
|
||||
.. _parsel changelog: https://parsel.readthedocs.io/en/latest/history.html
|
||||
|
||||
Telnet console
|
||||
~~~~~~~~~~~~~~
|
||||
|
||||
**Backwards incompatible**: Scrapy's telnet console now requires username
|
||||
and password. See :ref:`topics-telnetconsole` for more details. This change
|
||||
fixes a **security issue**; see :ref:`release-1.5.2` release notes for details.
|
||||
|
||||
New extensibility features
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
* ``from_crawler`` support is added to feed exporters and feed storages. This,
|
||||
among other things, allows to access Scrapy settings from custom feed
|
||||
storages and exporters (:issue:`1605`, :issue:`3348`).
|
||||
* ``from_crawler`` support is added to dupefilters (:issue:`2956`); this allows
|
||||
to access e.g. settings or a spider from a dupefilter.
|
||||
* :signal:`item_error` is fired when an error happens in a pipeline
|
||||
(:issue:`3256`);
|
||||
* :signal:`request_reached_downloader` is fired when Downloader gets
|
||||
a new Request; this signal can be useful e.g. for custom Schedulers
|
||||
(:issue:`3393`).
|
||||
* new SitemapSpider :meth:`~.SitemapSpider.sitemap_filter` method which allows
|
||||
to select sitemap entries based on their attributes in SitemapSpider
|
||||
subclasses (:issue:`3512`).
|
||||
* Lazy loading of Downloader Handlers is now optional; this enables better
|
||||
initialization error handling in custom Downloader Handlers (:issue:`3394`).
|
||||
|
||||
New FilePipeline and MediaPipeline features
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
* Expose more options for S3FilesStore: :setting:`AWS_ENDPOINT_URL`,
|
||||
:setting:`AWS_USE_SSL`, :setting:`AWS_VERIFY`, :setting:`AWS_REGION_NAME`.
|
||||
For example, this allows to use alternative or self-hosted
|
||||
AWS-compatible providers (:issue:`2609`, :issue:`3548`).
|
||||
* ACL support for Google Cloud Storage: :setting:`FILES_STORE_GCS_ACL` and
|
||||
:setting:`IMAGES_STORE_GCS_ACL` (:issue:`3199`).
|
||||
|
||||
``scrapy.contracts`` improvements
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
* Exceptions in contracts code are handled better (:issue:`3377`);
|
||||
* ``dont_filter=True`` is used for contract requests, which allows to test
|
||||
different callbacks with the same URL (:issue:`3381`);
|
||||
* ``request_cls`` attribute in Contract subclasses allow to use different
|
||||
Request classes in contracts, for example FormRequest (:issue:`3383`).
|
||||
* Fixed errback handling in contracts, e.g. for cases where a contract
|
||||
is executed for URL which returns non-200 response (:issue:`3371`).
|
||||
|
||||
Usability improvements
|
||||
~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
* more stats for RobotsTxtMiddleware (:issue:`3100`)
|
||||
* INFO log level is used to show telnet host/port (:issue:`3115`)
|
||||
* a message is added to IgnoreRequest in RobotsTxtMiddleware (:issue:`3113`)
|
||||
* better validation of ``url`` argument in ``Response.follow`` (:issue:`3131`)
|
||||
* non-zero exit code is returned from Scrapy commands when error happens
|
||||
on spider inititalization (:issue:`3226`)
|
||||
* Link extraction improvements: "ftp" is added to scheme list (:issue:`3152`);
|
||||
"flv" is added to common video extensions (:issue:`3165`)
|
||||
* better error message when an exporter is disabled (:issue:`3358`);
|
||||
* ``scrapy shell --help`` mentions syntax required for local files
|
||||
(``./file.html``) - :issue:`3496`.
|
||||
* Referer header value is added to RFPDupeFilter log messages (:issue:`3588`)
|
||||
|
||||
Bug fixes
|
||||
~~~~~~~~~
|
||||
|
||||
* fixed issue with extra blank lines in .csv exports under Windows
|
||||
(:issue:`3039`);
|
||||
* proper handling of pickling errors in Python 3 when serializing objects
|
||||
for disk queues (:issue:`3082`)
|
||||
* flags are now preserved when copying Requests (:issue:`3342`);
|
||||
* FormRequest.from_response clickdata shouldn't ignore elements with
|
||||
``input[type=image]`` (:issue:`3153`).
|
||||
* FormRequest.from_response should preserve duplicate keys (:issue:`3247`)
|
||||
|
||||
Documentation improvements
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
* Docs are re-written to suggest .get/.getall API instead of
|
||||
.extract/.extract_first. Also, :ref:`topics-selectors` docs are updated
|
||||
and re-structured to match latest parsel docs; they now contain more topics,
|
||||
such as :ref:`selecting-attributes` or :ref:`topics-selectors-css-extensions`
|
||||
(:issue:`3390`).
|
||||
* :ref:`topics-developer-tools` is a new tutorial which replaces
|
||||
old Firefox and Firebug tutorials (:issue:`3400`).
|
||||
* SCRAPY_PROJECT environment variable is documented (:issue:`3518`);
|
||||
* troubleshooting section is added to install instructions (:issue:`3517`);
|
||||
* improved links to beginner resources in the tutorial
|
||||
(:issue:`3367`, :issue:`3468`);
|
||||
* fixed :setting:`RETRY_HTTP_CODES` default values in docs (:issue:`3335`);
|
||||
* remove unused `DEPTH_STATS` option from docs (:issue:`3245`);
|
||||
* other cleanups (:issue:`3347`, :issue:`3350`, :issue:`3445`, :issue:`3544`,
|
||||
:issue:`3605`).
|
||||
|
||||
Deprecation removals
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Compatibility shims for pre-1.0 Scrapy module names are removed
|
||||
(:issue:`3318`):
|
||||
|
||||
* ``scrapy.command``
|
||||
* ``scrapy.contrib`` (with all submodules)
|
||||
* ``scrapy.contrib_exp`` (with all submodules)
|
||||
* ``scrapy.dupefilter``
|
||||
* ``scrapy.linkextractor``
|
||||
* ``scrapy.project``
|
||||
* ``scrapy.spider``
|
||||
* ``scrapy.spidermanager``
|
||||
* ``scrapy.squeue``
|
||||
* ``scrapy.stats``
|
||||
* ``scrapy.statscol``
|
||||
* ``scrapy.utils.decorator``
|
||||
|
||||
See :ref:`module-relocations` for more information, or use suggestions
|
||||
from Scrapy 1.5.x deprecation warnings to update your code.
|
||||
|
||||
Other deprecation removals:
|
||||
|
||||
* Deprecated scrapy.interfaces.ISpiderManager is removed; please use
|
||||
scrapy.interfaces.ISpiderLoader.
|
||||
* Deprecated ``CrawlerSettings`` class is removed (:issue:`3327`).
|
||||
* Deprecated ``Settings.overrides`` and ``Settings.defaults`` attributes
|
||||
are removed (:issue:`3327`, :issue:`3359`).
|
||||
|
||||
Other improvements, cleanups
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
* All Scrapy tests now pass on Windows; Scrapy testing suite is executed
|
||||
in a Windows environment on CI (:issue:`3315`).
|
||||
* Python 3.7 support (:issue:`3326`, :issue:`3150`, :issue:`3547`).
|
||||
* Testing and CI fixes (:issue:`3526`, :issue:`3538`, :issue:`3308`,
|
||||
:issue:`3311`, :issue:`3309`, :issue:`3305`, :issue:`3210`, :issue:`3299`)
|
||||
* ``scrapy.http.cookies.CookieJar.clear`` accepts "domain", "path" and "name"
|
||||
optional arguments (:issue:`3231`).
|
||||
* additional files are included to sdist (:issue:`3495`);
|
||||
* code style fixes (:issue:`3405`, :issue:`3304`);
|
||||
* unneeded .strip() call is removed (:issue:`3519`);
|
||||
* collections.deque is used to store MiddlewareManager methods instead
|
||||
of a list (:issue:`3476`)
|
||||
|
||||
.. _release-1.5.2:
|
||||
|
||||
Scrapy 1.5.2 (2019-01-22)
|
||||
-------------------------
|
||||
|
|
@ -29,6 +218,8 @@ Scrapy 1.5.2 (2019-01-22)
|
|||
|
||||
* Backport CI build failure under GCE environemnt due to boto import error.
|
||||
|
||||
.. _release-1.5.1:
|
||||
|
||||
Scrapy 1.5.1 (2018-07-12)
|
||||
-------------------------
|
||||
|
||||
|
|
@ -44,6 +235,9 @@ This is a maintenance release with important bug fixes, but no new features:
|
|||
:issue:`3279`, :issue:`3201`, :issue:`3260`, :issue:`3284`, :issue:`3298`,
|
||||
:issue:`3294`).
|
||||
|
||||
|
||||
.. _release-1.5.0:
|
||||
|
||||
Scrapy 1.5.0 (2017-12-29)
|
||||
-------------------------
|
||||
|
||||
|
|
@ -155,6 +349,7 @@ Docs
|
|||
- Document ``from_crawler`` methods for spider and downloader middlewares
|
||||
(:issue:`3019`)
|
||||
|
||||
.. _release-1.4.0:
|
||||
|
||||
Scrapy 1.4.0 (2017-05-18)
|
||||
-------------------------
|
||||
|
|
@ -341,6 +536,8 @@ Documentation
|
|||
- Clarify ``allowed_domains`` example (:issue:`2670`)
|
||||
|
||||
|
||||
.. _release-1.3.3:
|
||||
|
||||
Scrapy 1.3.3 (2017-03-10)
|
||||
-------------------------
|
||||
|
||||
|
|
@ -353,6 +550,7 @@ Bug fixes
|
|||
A new setting is introduced to toggle between warning or exception if needed ;
|
||||
see :setting:`SPIDER_LOADER_WARN_ONLY` for details.
|
||||
|
||||
.. _release-1.3.2:
|
||||
|
||||
Scrapy 1.3.2 (2017-02-13)
|
||||
-------------------------
|
||||
|
|
@ -364,6 +562,8 @@ Bug fixes
|
|||
- Use consistent selectors for author field in tutorial (:issue:`2551`).
|
||||
- Fix TLS compatibility in Twisted 17+ (:issue:`2558`)
|
||||
|
||||
.. _release-1.3.1:
|
||||
|
||||
Scrapy 1.3.1 (2017-02-08)
|
||||
-------------------------
|
||||
|
||||
|
|
@ -412,6 +612,8 @@ Cleanups
|
|||
- Remove dead code supporting old Twisted versions (:issue:`2544`).
|
||||
|
||||
|
||||
.. _release-1.3.0:
|
||||
|
||||
Scrapy 1.3.0 (2016-12-21)
|
||||
-------------------------
|
||||
|
||||
|
|
@ -451,6 +653,7 @@ Dependencies & Cleanups
|
|||
- ``ChunkedTransferMiddleware`` is deprecated and removed from the default
|
||||
downloader middlewares.
|
||||
|
||||
.. _release-1.2.3:
|
||||
|
||||
Scrapy 1.2.3 (2017-03-03)
|
||||
-------------------------
|
||||
|
|
@ -458,6 +661,8 @@ Scrapy 1.2.3 (2017-03-03)
|
|||
- Packaging fix: disallow unsupported Twisted versions in setup.py
|
||||
|
||||
|
||||
.. _release-1.2.2:
|
||||
|
||||
Scrapy 1.2.2 (2016-12-06)
|
||||
-------------------------
|
||||
|
||||
|
|
@ -493,6 +698,8 @@ Other changes
|
|||
.. _conda-forge: https://anaconda.org/conda-forge/scrapy
|
||||
|
||||
|
||||
.. _release-1.2.1:
|
||||
|
||||
Scrapy 1.2.1 (2016-10-21)
|
||||
-------------------------
|
||||
|
||||
|
|
@ -517,6 +724,8 @@ Other changes
|
|||
- Removed ``www.`` from ``start_urls`` in built-in spider templates (:issue:`2299`).
|
||||
|
||||
|
||||
.. _release-1.2.0:
|
||||
|
||||
Scrapy 1.2.0 (2016-10-03)
|
||||
-------------------------
|
||||
|
||||
|
|
@ -585,12 +794,14 @@ Documentation
|
|||
- Reworded misleading :setting:`RANDOMIZE_DOWNLOAD_DELAY` description (:issue:`2190`).
|
||||
- Add StackOverflow as a support channel (:issue:`2257`).
|
||||
|
||||
.. _release-1.1.4:
|
||||
|
||||
Scrapy 1.1.4 (2017-03-03)
|
||||
-------------------------
|
||||
|
||||
- Packaging fix: disallow unsupported Twisted versions in setup.py
|
||||
|
||||
.. _release-1.1.3:
|
||||
|
||||
Scrapy 1.1.3 (2016-09-22)
|
||||
-------------------------
|
||||
|
|
@ -608,6 +819,7 @@ Documentation
|
|||
rewritten to use http://toscrape.com websites
|
||||
(:issue:`2236`, :issue:`2249`, :issue:`2252`).
|
||||
|
||||
.. _release-1.1.2:
|
||||
|
||||
Scrapy 1.1.2 (2016-08-18)
|
||||
-------------------------
|
||||
|
|
@ -622,6 +834,7 @@ Bug fixes
|
|||
- :setting:`IMAGES_EXPIRES` default value set back to 90
|
||||
(the regression was introduced in 1.1.1)
|
||||
|
||||
.. _release-1.1.1:
|
||||
|
||||
Scrapy 1.1.1 (2016-07-13)
|
||||
-------------------------
|
||||
|
|
@ -674,6 +887,7 @@ Tests
|
|||
|
||||
- Upgrade py.test requirement on Travis CI and Pin pytest-cov to 2.2.1 (:issue:`2095`)
|
||||
|
||||
.. _release-1.1.0:
|
||||
|
||||
Scrapy 1.1.0 (2016-05-11)
|
||||
-------------------------
|
||||
|
|
@ -863,12 +1077,14 @@ Bugfixes
|
|||
- HTTPS+CONNECT tunnels could get mixed up when using multiple proxies
|
||||
to same remote host (:issue:`1912`).
|
||||
|
||||
.. _release-1.0.7:
|
||||
|
||||
Scrapy 1.0.7 (2017-03-03)
|
||||
-------------------------
|
||||
|
||||
- Packaging fix: disallow unsupported Twisted versions in setup.py
|
||||
|
||||
.. _release-1.0.6:
|
||||
|
||||
Scrapy 1.0.6 (2016-05-04)
|
||||
-------------------------
|
||||
|
|
@ -878,6 +1094,7 @@ Scrapy 1.0.6 (2016-05-04)
|
|||
- DOC: Support for Sphinx 1.4+ (:issue:`1893`)
|
||||
- DOC: Consistency in selectors examples (:issue:`1869`)
|
||||
|
||||
.. _release-1.0.5:
|
||||
|
||||
Scrapy 1.0.5 (2016-02-04)
|
||||
-------------------------
|
||||
|
|
@ -887,6 +1104,7 @@ Scrapy 1.0.5 (2016-02-04)
|
|||
- DOC: Fixed typos in tutorial and media-pipeline (:commit:`808a9ea` and :commit:`803bd87`)
|
||||
- DOC: Add AjaxCrawlMiddleware to DOWNLOADER_MIDDLEWARES_BASE in settings docs (:commit:`aa94121`)
|
||||
|
||||
.. _release-1.0.4:
|
||||
|
||||
Scrapy 1.0.4 (2015-12-30)
|
||||
-------------------------
|
||||
|
|
@ -940,12 +1158,16 @@ Scrapy 1.0.4 (2015-12-30)
|
|||
- Small grammatical change (:commit:`8752294`)
|
||||
- Add openssl version to version command (:commit:`13c45ac`)
|
||||
|
||||
.. _release-1.0.3:
|
||||
|
||||
Scrapy 1.0.3 (2015-08-11)
|
||||
-------------------------
|
||||
|
||||
- add service_identity to scrapy install_requires (:commit:`cbc2501`)
|
||||
- Workaround for travis#296 (:commit:`66af9cd`)
|
||||
|
||||
.. _release-1.0.2:
|
||||
|
||||
Scrapy 1.0.2 (2015-08-06)
|
||||
-------------------------
|
||||
|
||||
|
|
@ -956,6 +1178,8 @@ Scrapy 1.0.2 (2015-08-06)
|
|||
- Fixed typos (:commit:`a9ae7b0`)
|
||||
- Fix doc reference. (:commit:`7c8a4fe`)
|
||||
|
||||
.. _release-1.0.1:
|
||||
|
||||
Scrapy 1.0.1 (2015-07-01)
|
||||
-------------------------
|
||||
|
||||
|
|
@ -966,6 +1190,8 @@ Scrapy 1.0.1 (2015-07-01)
|
|||
- DOC remove version suffix from ubuntu package (:commit:`5303c66`)
|
||||
- DOC Update release date for 1.0 (:commit:`c89fa29`)
|
||||
|
||||
.. _release-1.0.0:
|
||||
|
||||
Scrapy 1.0.0 (2015-06-19)
|
||||
-------------------------
|
||||
|
||||
|
|
@ -1080,6 +1306,8 @@ until it reaches a stable status.
|
|||
|
||||
See more examples for scripts running Scrapy: :ref:`topics-practices`
|
||||
|
||||
.. _module-relocations:
|
||||
|
||||
Module Relocations
|
||||
~~~~~~~~~~~~~~~~~~
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue