mirror of https://github.com/scrapy/scrapy.git
Cover Scrapy 2.2.1 and 2.3 in the release notes (#4708)
This commit is contained in:
Adrián Chaves
GitHub
parent
015b71d89f
commit
3600582f56
133
docs/news.rst
133
docs/news.rst
|
|
@ -3,6 +3,139 @@
|
|||
Release notes
|
||||
=============
|
||||
|
||||
.. _release-2.3.0:
|
||||
|
||||
Scrapy 2.3.0 (2020-08-04)
|
||||
-------------------------
|
||||
|
||||
Highlights:
|
||||
|
||||
* :ref:`Feed exports <topics-feed-exports>` now support :ref:`Google Cloud
|
||||
Storage <topics-feed-storage-gcs>` as a storage backend
|
||||
|
||||
* The new :setting:`FEED_EXPORT_BATCH_ITEM_COUNT` setting allows to deliver
|
||||
output items in batches of up to the specified number of items.
|
||||
|
||||
It also serves as a workaround for :ref:`delayed file delivery
|
||||
<delayed-file-delivery>`, which causes Scrapy to only start item delivery
|
||||
after the crawl has finished when using certain storage backends
|
||||
(:ref:`S3 <topics-feed-storage-s3>`, :ref:`FTP <topics-feed-storage-ftp>`,
|
||||
and now :ref:`GCS <topics-feed-storage-gcs>`).
|
||||
|
||||
* The base implementation of :ref:`item loaders <topics-loaders>` has been
|
||||
moved into a separate library, :doc:`itemloaders <itemloaders:index>`,
|
||||
allowing usage from outside Scrapy and a separate release schedule
|
||||
|
||||
Deprecation removals
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
* Removed the following classes and their parent modules from
|
||||
``scrapy.linkextractors``:
|
||||
|
||||
* ``htmlparser.HtmlParserLinkExtractor``
|
||||
* ``regex.RegexLinkExtractor``
|
||||
* ``sgml.BaseSgmlLinkExtractor``
|
||||
* ``sgml.SgmlLinkExtractor``
|
||||
|
||||
Use
|
||||
:class:`LinkExtractor <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>`
|
||||
instead (:issue:`4356`, :issue:`4679`)
|
||||
|
||||
|
||||
Deprecations
|
||||
~~~~~~~~~~~~
|
||||
|
||||
* The ``scrapy.utils.python.retry_on_eintr`` function is now deprecated
|
||||
(:issue:`4683`)
|
||||
|
||||
|
||||
New features
|
||||
~~~~~~~~~~~~
|
||||
|
||||
* :ref:`Feed exports <topics-feed-exports>` support :ref:`Google Cloud
|
||||
Storage <topics-feed-storage-gcs>` (:issue:`685`, :issue:`3608`)
|
||||
|
||||
* New :setting:`FEED_EXPORT_BATCH_ITEM_COUNT` setting for batch deliveries
|
||||
(:issue:`4250`, :issue:`4434`)
|
||||
|
||||
* The :command:`parse` command now allows specifying an output file
|
||||
(:issue:`4317`, :issue:`4377`)
|
||||
|
||||
* :meth:`Request.from_curl <scrapy.http.Request.from_curl>` and
|
||||
:func:`~scrapy.utils.curl.curl_to_request_kwargs` now also support
|
||||
``--data-raw`` (:issue:`4612`)
|
||||
|
||||
* A ``parse`` callback may now be used in built-in spider subclasses, such
|
||||
as :class:`~scrapy.spiders.CrawlSpider` (:issue:`712`, :issue:`732`,
|
||||
:issue:`781`, :issue:`4254` )
|
||||
|
||||
|
||||
Bug fixes
|
||||
~~~~~~~~~
|
||||
|
||||
* Fixed the :ref:`CSV exporting <topics-feed-format-csv>` of
|
||||
:ref:`dataclass items <dataclass-items>` and :ref:`attr.s items
|
||||
<attrs-items>` (:issue:`4667`, :issue:`4668`)
|
||||
|
||||
* :meth:`Request.from_curl <scrapy.http.Request.from_curl>` and
|
||||
:func:`~scrapy.utils.curl.curl_to_request_kwargs` now set the request
|
||||
method to ``POST`` when a request body is specified and no request method
|
||||
is specified (:issue:`4612`)
|
||||
|
||||
* The processing of ANSI escape sequences in enabled in Windows 10.0.14393
|
||||
and later, where it is required for colored output (:issue:`4393`,
|
||||
:issue:`4403`)
|
||||
|
||||
|
||||
Documentation
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
* Updated the `OpenSSL cipher list format`_ link in the documentation about
|
||||
the :setting:`DOWNLOADER_CLIENT_TLS_CIPHERS` setting (:issue:`4653`)
|
||||
|
||||
* Simplified the code example in :ref:`topics-loaders-dataclass`
|
||||
(:issue:`4652`)
|
||||
|
||||
.. _OpenSSL cipher list format: https://www.openssl.org/docs/manmaster/man1/openssl-ciphers.html#CIPHER-LIST-FORMAT
|
||||
|
||||
|
||||
Quality assurance
|
||||
~~~~~~~~~~~~~~~~~
|
||||
|
||||
* The base implementation of :ref:`item loaders <topics-loaders>` has been
|
||||
moved into :doc:`itemloaders <itemloaders:index>` (:issue:`4005`,
|
||||
:issue:`4516`)
|
||||
|
||||
* Fixed a silenced error in some scheduler tests (:issue:`4644`,
|
||||
:issue:`4645`)
|
||||
|
||||
* Renewed the localhost certificate used for SSL tests (:issue:`4650`)
|
||||
|
||||
* Removed cookie-handling code specific to Python 2 (:issue:`4682`)
|
||||
|
||||
* Stopped using Python 2 unicode literal syntax (:issue:`4704`)
|
||||
|
||||
* Stopped using a backlash for line continuation (:issue:`4673`)
|
||||
|
||||
* Removed unneeded entries from the MyPy exception list (:issue:`4690`)
|
||||
|
||||
* Automated tests now pass on Windows as part of our continuous integration
|
||||
system (:issue:`4458`)
|
||||
|
||||
* Automated tests now pass on the latest PyPy version for supported Python
|
||||
versions in our continuous integration system (:issue:`4504`)
|
||||
|
||||
|
||||
.. _release-2.2.1:
|
||||
|
||||
Scrapy 2.2.1 (2020-07-17)
|
||||
-------------------------
|
||||
|
||||
* The :command:`startproject` command no longer makes unintended changes to
|
||||
the permissions of files in the destination folder, such as removing
|
||||
execution permissions (:issue:`4662`, :issue:`4666`)
|
||||
|
||||
|
||||
.. _release-2.2.0:
|
||||
|
||||
Scrapy 2.2.0 (2020-06-24)
|
||||
|
|
|
|||
|
|
@ -493,6 +493,8 @@ Supported options:
|
|||
|
||||
* ``--output`` or ``-o``: dump scraped items to a file
|
||||
|
||||
.. versionadded:: 2.3
|
||||
|
||||
.. skip: start
|
||||
|
||||
Usage example::
|
||||
|
|
|
|||
|
|
@ -289,8 +289,10 @@ request::
|
|||
"://quotes.toscrape.com/scroll' -H 'Cache-Control: max-age=0'")
|
||||
|
||||
Alternatively, if you want to know the arguments needed to recreate that
|
||||
request you can use the :func:`scrapy.utils.curl.curl_to_request_kwargs`
|
||||
function to get a dictionary with the equivalent arguments.
|
||||
request you can use the :func:`~scrapy.utils.curl.curl_to_request_kwargs`
|
||||
function to get a dictionary with the equivalent arguments:
|
||||
|
||||
.. autofunction:: scrapy.utils.curl.curl_to_request_kwargs
|
||||
|
||||
Note that to translate a cURL command into a Scrapy request,
|
||||
you may use `curl2scrapy <https://michael-shub.github.io/curl2scrapy/>`_.
|
||||
|
|
|
|||
|
|
@ -100,6 +100,7 @@ The storages backends supported out of the box are:
|
|||
* :ref:`topics-feed-storage-fs`
|
||||
* :ref:`topics-feed-storage-ftp`
|
||||
* :ref:`topics-feed-storage-s3` (requires botocore_)
|
||||
* :ref:`topics-feed-storage-gcs` (requires `google-cloud-storage`_)
|
||||
* :ref:`topics-feed-storage-stdout`
|
||||
|
||||
Some storage backends may be unavailable if the required external libraries are
|
||||
|
|
@ -169,6 +170,9 @@ FTP supports two different connection modes: `active or passive
|
|||
mode by default. To use the active connection mode instead, set the
|
||||
:setting:`FEED_STORAGE_FTP_ACTIVE` setting to ``True``.
|
||||
|
||||
This storage backend uses :ref:`delayed file delivery <delayed-file-delivery>`.
|
||||
|
||||
|
||||
.. _topics-feed-storage-s3:
|
||||
|
||||
S3
|
||||
|
|
@ -194,11 +198,16 @@ You can also define a custom ACL for exported feeds using this setting:
|
|||
|
||||
* :setting:`FEED_STORAGE_S3_ACL`
|
||||
|
||||
This storage backend uses :ref:`delayed file delivery <delayed-file-delivery>`.
|
||||
|
||||
|
||||
.. _topics-feed-storage-gcs:
|
||||
|
||||
Google Cloud Storage (GCS)
|
||||
--------------------------
|
||||
|
||||
.. versionadded:: 2.3
|
||||
|
||||
The feeds are stored on `Google Cloud Storage`_.
|
||||
|
||||
* URI scheme: ``gs``
|
||||
|
|
@ -206,7 +215,7 @@ The feeds are stored on `Google Cloud Storage`_.
|
|||
|
||||
* ``gs://mybucket/path/to/export.csv``
|
||||
|
||||
* Required external libraries: `google-cloud-storage <https://cloud.google.com/storage/docs/reference/libraries#client-libraries-install-python>`_.
|
||||
* Required external libraries: `google-cloud-storage`_.
|
||||
|
||||
For more information about authentication, please refer to `Google Cloud documentation <https://cloud.google.com/docs/authentication/production>`_.
|
||||
|
||||
|
|
@ -215,6 +224,11 @@ You can set a *Project ID* and *Access Control List (ACL)* through the following
|
|||
* :setting:`FEED_STORAGE_GCS_ACL`
|
||||
* :setting:`GCS_PROJECT_ID`
|
||||
|
||||
This storage backend uses :ref:`delayed file delivery <delayed-file-delivery>`.
|
||||
|
||||
.. _google-cloud-storage: https://cloud.google.com/storage/docs/reference/libraries#client-libraries-install-python
|
||||
|
||||
|
||||
.. _topics-feed-storage-stdout:
|
||||
|
||||
Standard output
|
||||
|
|
@ -227,6 +241,26 @@ The feeds are written to the standard output of the Scrapy process.
|
|||
* Required external libraries: none
|
||||
|
||||
|
||||
.. _delayed-file-delivery:
|
||||
|
||||
Delayed file delivery
|
||||
---------------------
|
||||
|
||||
As indicated above, some of the described storage backends use delayed file
|
||||
delivery.
|
||||
|
||||
These storage backends do not upload items to the feed URI as those items are
|
||||
scraped. Instead, Scrapy writes items into a temporary local file, and only
|
||||
once all the file contents have been written (i.e. at the end of the crawl) is
|
||||
that file uploaded to the feed URI.
|
||||
|
||||
If you want item delivery to start earlier when using one of these storage
|
||||
backends, use :setting:`FEED_EXPORT_BATCH_ITEM_COUNT` to split the output items
|
||||
in multiple files, with the specified maximum item count per file. That way, as
|
||||
soon as a file reaches the maximum item count, that file is delivered to the
|
||||
feed URI, allowing item delivery to start way before the end of the crawl.
|
||||
|
||||
|
||||
Settings
|
||||
========
|
||||
|
||||
|
|
|
|||
|
|
@ -39,7 +39,8 @@ def curl_to_request_kwargs(curl_command, ignore_unknown_options=True):
|
|||
|
||||
:param str curl_command: string containing the curl command
|
||||
:param bool ignore_unknown_options: If true, only a warning is emitted when
|
||||
cURL options are unknown. Otherwise raises an error. (default: True)
|
||||
cURL options are unknown. Otherwise
|
||||
raises an error. (default: True)
|
||||
:return: dictionary of Request kwargs
|
||||
"""
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue