diff --git a/docs/news.rst b/docs/news.rst index 80d130e4a..850b323ef 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -3,6 +3,139 @@ Release notes ============= +.. _release-2.3.0: + +Scrapy 2.3.0 (2020-08-04) +------------------------- + +Highlights: + +* :ref:`Feed exports ` now support :ref:`Google Cloud + Storage ` as a storage backend + +* The new :setting:`FEED_EXPORT_BATCH_ITEM_COUNT` setting allows to deliver + output items in batches of up to the specified number of items. + + It also serves as a workaround for :ref:`delayed file delivery + `, which causes Scrapy to only start item delivery + after the crawl has finished when using certain storage backends + (:ref:`S3 `, :ref:`FTP `, + and now :ref:`GCS `). + +* The base implementation of :ref:`item loaders ` has been + moved into a separate library, :doc:`itemloaders `, + allowing usage from outside Scrapy and a separate release schedule + +Deprecation removals +~~~~~~~~~~~~~~~~~~~~ + +* Removed the following classes and their parent modules from + ``scrapy.linkextractors``: + + * ``htmlparser.HtmlParserLinkExtractor`` + * ``regex.RegexLinkExtractor`` + * ``sgml.BaseSgmlLinkExtractor`` + * ``sgml.SgmlLinkExtractor`` + + Use + :class:`LinkExtractor ` + instead (:issue:`4356`, :issue:`4679`) + + +Deprecations +~~~~~~~~~~~~ + +* The ``scrapy.utils.python.retry_on_eintr`` function is now deprecated + (:issue:`4683`) + + +New features +~~~~~~~~~~~~ + +* :ref:`Feed exports ` support :ref:`Google Cloud + Storage ` (:issue:`685`, :issue:`3608`) + +* New :setting:`FEED_EXPORT_BATCH_ITEM_COUNT` setting for batch deliveries + (:issue:`4250`, :issue:`4434`) + +* The :command:`parse` command now allows specifying an output file + (:issue:`4317`, :issue:`4377`) + +* :meth:`Request.from_curl ` and + :func:`~scrapy.utils.curl.curl_to_request_kwargs` now also support + ``--data-raw`` (:issue:`4612`) + +* A ``parse`` callback may now be used in built-in spider subclasses, such + as :class:`~scrapy.spiders.CrawlSpider` (:issue:`712`, :issue:`732`, + :issue:`781`, :issue:`4254` ) + + +Bug fixes +~~~~~~~~~ + +* Fixed the :ref:`CSV exporting ` of + :ref:`dataclass items ` and :ref:`attr.s items + ` (:issue:`4667`, :issue:`4668`) + +* :meth:`Request.from_curl ` and + :func:`~scrapy.utils.curl.curl_to_request_kwargs` now set the request + method to ``POST`` when a request body is specified and no request method + is specified (:issue:`4612`) + +* The processing of ANSI escape sequences in enabled in Windows 10.0.14393 + and later, where it is required for colored output (:issue:`4393`, + :issue:`4403`) + + +Documentation +~~~~~~~~~~~~~ + +* Updated the `OpenSSL cipher list format`_ link in the documentation about + the :setting:`DOWNLOADER_CLIENT_TLS_CIPHERS` setting (:issue:`4653`) + +* Simplified the code example in :ref:`topics-loaders-dataclass` + (:issue:`4652`) + +.. _OpenSSL cipher list format: https://www.openssl.org/docs/manmaster/man1/openssl-ciphers.html#CIPHER-LIST-FORMAT + + +Quality assurance +~~~~~~~~~~~~~~~~~ + +* The base implementation of :ref:`item loaders ` has been + moved into :doc:`itemloaders ` (:issue:`4005`, + :issue:`4516`) + +* Fixed a silenced error in some scheduler tests (:issue:`4644`, + :issue:`4645`) + +* Renewed the localhost certificate used for SSL tests (:issue:`4650`) + +* Removed cookie-handling code specific to Python 2 (:issue:`4682`) + +* Stopped using Python 2 unicode literal syntax (:issue:`4704`) + +* Stopped using a backlash for line continuation (:issue:`4673`) + +* Removed unneeded entries from the MyPy exception list (:issue:`4690`) + +* Automated tests now pass on Windows as part of our continuous integration + system (:issue:`4458`) + +* Automated tests now pass on the latest PyPy version for supported Python + versions in our continuous integration system (:issue:`4504`) + + +.. _release-2.2.1: + +Scrapy 2.2.1 (2020-07-17) +------------------------- + +* The :command:`startproject` command no longer makes unintended changes to + the permissions of files in the destination folder, such as removing + execution permissions (:issue:`4662`, :issue:`4666`) + + .. _release-2.2.0: Scrapy 2.2.0 (2020-06-24) diff --git a/docs/topics/commands.rst b/docs/topics/commands.rst index 4fce51abc..9638a2322 100644 --- a/docs/topics/commands.rst +++ b/docs/topics/commands.rst @@ -493,6 +493,8 @@ Supported options: * ``--output`` or ``-o``: dump scraped items to a file + .. versionadded:: 2.3 + .. skip: start Usage example:: diff --git a/docs/topics/developer-tools.rst b/docs/topics/developer-tools.rst index 4e87a00f2..101aa159c 100644 --- a/docs/topics/developer-tools.rst +++ b/docs/topics/developer-tools.rst @@ -289,8 +289,10 @@ request:: "://quotes.toscrape.com/scroll' -H 'Cache-Control: max-age=0'") Alternatively, if you want to know the arguments needed to recreate that -request you can use the :func:`scrapy.utils.curl.curl_to_request_kwargs` -function to get a dictionary with the equivalent arguments. +request you can use the :func:`~scrapy.utils.curl.curl_to_request_kwargs` +function to get a dictionary with the equivalent arguments: + +.. autofunction:: scrapy.utils.curl.curl_to_request_kwargs Note that to translate a cURL command into a Scrapy request, you may use `curl2scrapy `_. diff --git a/docs/topics/feed-exports.rst b/docs/topics/feed-exports.rst index dd4eb3c61..37b7096f6 100644 --- a/docs/topics/feed-exports.rst +++ b/docs/topics/feed-exports.rst @@ -100,6 +100,7 @@ The storages backends supported out of the box are: * :ref:`topics-feed-storage-fs` * :ref:`topics-feed-storage-ftp` * :ref:`topics-feed-storage-s3` (requires botocore_) + * :ref:`topics-feed-storage-gcs` (requires `google-cloud-storage`_) * :ref:`topics-feed-storage-stdout` Some storage backends may be unavailable if the required external libraries are @@ -169,6 +170,9 @@ FTP supports two different connection modes: `active or passive mode by default. To use the active connection mode instead, set the :setting:`FEED_STORAGE_FTP_ACTIVE` setting to ``True``. +This storage backend uses :ref:`delayed file delivery `. + + .. _topics-feed-storage-s3: S3 @@ -194,11 +198,16 @@ You can also define a custom ACL for exported feeds using this setting: * :setting:`FEED_STORAGE_S3_ACL` +This storage backend uses :ref:`delayed file delivery `. + + .. _topics-feed-storage-gcs: Google Cloud Storage (GCS) -------------------------- +.. versionadded:: 2.3 + The feeds are stored on `Google Cloud Storage`_. * URI scheme: ``gs`` @@ -206,7 +215,7 @@ The feeds are stored on `Google Cloud Storage`_. * ``gs://mybucket/path/to/export.csv`` - * Required external libraries: `google-cloud-storage `_. + * Required external libraries: `google-cloud-storage`_. For more information about authentication, please refer to `Google Cloud documentation `_. @@ -215,6 +224,11 @@ You can set a *Project ID* and *Access Control List (ACL)* through the following * :setting:`FEED_STORAGE_GCS_ACL` * :setting:`GCS_PROJECT_ID` +This storage backend uses :ref:`delayed file delivery `. + +.. _google-cloud-storage: https://cloud.google.com/storage/docs/reference/libraries#client-libraries-install-python + + .. _topics-feed-storage-stdout: Standard output @@ -227,6 +241,26 @@ The feeds are written to the standard output of the Scrapy process. * Required external libraries: none +.. _delayed-file-delivery: + +Delayed file delivery +--------------------- + +As indicated above, some of the described storage backends use delayed file +delivery. + +These storage backends do not upload items to the feed URI as those items are +scraped. Instead, Scrapy writes items into a temporary local file, and only +once all the file contents have been written (i.e. at the end of the crawl) is +that file uploaded to the feed URI. + +If you want item delivery to start earlier when using one of these storage +backends, use :setting:`FEED_EXPORT_BATCH_ITEM_COUNT` to split the output items +in multiple files, with the specified maximum item count per file. That way, as +soon as a file reaches the maximum item count, that file is delivered to the +feed URI, allowing item delivery to start way before the end of the crawl. + + Settings ======== diff --git a/scrapy/utils/curl.py b/scrapy/utils/curl.py index aa681522f..9c0efcec4 100644 --- a/scrapy/utils/curl.py +++ b/scrapy/utils/curl.py @@ -39,7 +39,8 @@ def curl_to_request_kwargs(curl_command, ignore_unknown_options=True): :param str curl_command: string containing the curl command :param bool ignore_unknown_options: If true, only a warning is emitted when - cURL options are unknown. Otherwise raises an error. (default: True) + cURL options are unknown. Otherwise + raises an error. (default: True) :return: dictionary of Request kwargs """