RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Write release notes for Scrapy 2.0.0 (#4329)

This commit is contained in:
Adrián Chaves 2020-03-03 09:11:11 +01:00 committed by GitHub
parent ef00f8eb8e
commit 6aa0ba4553
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
21 changed files with 704 additions and 43 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
docs/index.rst
View File

@ -165,6 +165,8 @@ Solving specific problems
topics/autothrottle
topics/benchmarking
topics/jobs
topics/coroutines
topics/asyncio
:doc:`faq`
Get answers to most frequently asked questions.
@ -205,6 +207,12 @@ Solving specific problems
:doc:`topics/jobs`
Learn how to pause and resume crawls for large spiders.
:doc:`topics/coroutines`
Use the :ref:`coroutine syntax <async>`.
:doc:`topics/asyncio`
Use :mod:`asyncio` and :mod:`asyncio`-powered libraries.
.. _extending-scrapy:
Extending Scrapy

4
docs/intro/install.rst
View File

@ -7,8 +7,8 @@ Installation guide
Installing Scrapy
=================
Scrapy runs on Python 3.5 or above
under CPython (default Python implementation) and PyPy (starting with PyPy 5.9).
Scrapy runs on Python 3.5 or above under CPython (default Python
implementation) and PyPy (starting with PyPy 5.9).
If you're using `Anaconda`_ or `Miniconda`_, you can install the package from
the `conda-forge`_ channel, which has up-to-date packages for Linux, Windows

458
docs/news.rst
View File

@ -3,8 +3,452 @@
Release notes
=============
.. note:: Scrapy 1.x will be the last series supporting Python 2. Scrapy 2.0,
planned for Q4 2019 or Q1 2020, will support **Python 3 only**.
.. _release-2.0.0:
Scrapy 2.0.0 (2020-03-03)
-------------------------
Highlights:
* Python 2 support has been removed
* :doc:`Partial <topics/coroutines>` :ref:`coroutine syntax <async>` support
and :doc:`experimental <topics/asyncio>` :mod:`asyncio` support
* New :meth:`Response.follow_all <scrapy.http.Response.follow_all>` method
* :ref:`FTP support <media-pipeline-ftp>` for media pipelines
* New :attr:`Response.certificate <scrapy.http.Response.certificate>`
attribute
* IPv6 support through :setting:`DNS_RESOLVER`
Backward-incompatible changes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
* Python 2 support has been removed, following `Python 2 end-of-life on
January 1, 2020`_ (:issue:`4091`, :issue:`4114`, :issue:`4115`,
:issue:`4121`, :issue:`4138`, :issue:`4231`, :issue:`4242`, :issue:`4304`,
:issue:`4309`, :issue:`4373`)
* Retry gaveups (see :setting:`RETRY_TIMES`) are now logged as errors instead
of as debug information (:issue:`3171`, :issue:`3566`)
* File extensions that
:class:`LinkExtractor <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>`
ignores by default now also include ``7z``, ``7zip``, ``apk``, ``bz2``,
``cdr``, ``dmg``, ``ico``, ``iso``, ``tar``, ``tar.gz``, ``webm``, and
``xz`` (:issue:`1837`, :issue:`2067`, :issue:`4066`)
* The :setting:`METAREFRESH_IGNORE_TAGS` setting is now an empty list by
default, following web browser behavior (:issue:`3844`, :issue:`4311`)
* The
:class:`~scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware`
now includes spaces after commas in the value of the ``Accept-Encoding``
header that it sets, following web browser behavior (:issue:`4293`)
* The ``__init__`` method of custom download handlers (see
:setting:`DOWNLOAD_HANDLERS`) or subclasses of the following downloader
handlers no longer receives a ``settings`` parameter:
* :class:`scrapy.core.downloader.handlers.datauri.DataURIDownloadHandler`
* :class:`scrapy.core.downloader.handlers.file.FileDownloadHandler`
Use the ``from_settings`` or ``from_crawler`` class methods to expose such
a parameter to your custom download handlers.
(:issue:`4126`)
* We have refactored the :class:`scrapy.core.scheduler.Scheduler` class and
related queue classes (see :setting:`SCHEDULER_PRIORITY_QUEUE`,
:setting:`SCHEDULER_DISK_QUEUE` and :setting:`SCHEDULER_MEMORY_QUEUE`) to
make it easier to implement custom scheduler queue classes. See
:ref:`2-0-0-scheduler-queue-changes` below for details.
* Overridden settings are now logged in a different format. This is more in
line with similar information logged at startup (:issue:`4199`)
.. _Python 2 end-of-life on January 1, 2020: https://www.python.org/doc/sunset-python-2/
Deprecation removals
~~~~~~~~~~~~~~~~~~~~
* The :ref:`Scrapy shell <topics-shell>` no longer provides a `sel` proxy
object, use :meth:`response.selector <scrapy.http.Response.selector>`
instead (:issue:`4347`)
* LevelDB support has been removed (:issue:`4112`)
* The following functions have been removed from :mod:`scrapy.utils.python`:
``isbinarytext``, ``is_writable``, ``setattr_default``, ``stringify_dict``
(:issue:`4362`)
Deprecations
~~~~~~~~~~~~
* Using environment variables prefixed with ``SCRAPY_`` to override settings
is deprecated (:issue:`4300`, :issue:`4374`, :issue:`4375`)
* :class:`scrapy.linkextractors.FilteringLinkExtractor` is deprecated, use
:class:`scrapy.linkextractors.LinkExtractor
<scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>` instead (:issue:`4045`)
* The ``noconnect`` query string argument of proxy URLs is deprecated and
should be removed from proxy URLs (:issue:`4198`)
* The :meth:`next <scrapy.utils.python.MutableChain.next>` method of
:class:`scrapy.utils.python.MutableChain` is deprecated, use the global
:func:`next` function or :meth:`MutableChain.__next__
<scrapy.utils.python.MutableChain.__next__>` instead (:issue:`4153`)
New features
~~~~~~~~~~~~
* Added :doc:`partial support <topics/coroutines>` for Python’s
:ref:`coroutine syntax <async>` and :doc:`experimental support
<topics/asyncio>` for :mod:`asyncio` and :mod:`asyncio`-powered libraries
(:issue:`4010`, :issue:`4259`, :issue:`4269`, :issue:`4270`, :issue:`4271`,
:issue:`4316`, :issue:`4318`)
* The new :meth:`Response.follow_all <scrapy.http.Response.follow_all>`
method offers the same functionality as
:meth:`Response.follow <scrapy.http.Response.follow>` but supports an
iterable of URLs as input and returns an iterable of requests
(:issue:`2582`, :issue:`4057`, :issue:`4286`)
* :ref:`Media pipelines <topics-media-pipeline>` now support :ref:`FTP
storage <media-pipeline-ftp>` (:issue:`3928`, :issue:`3961`)
* The new :attr:`Response.certificate <scrapy.http.Response.certificate>`
attribute exposes the SSL certificate of the server as a
:class:`twisted.internet.ssl.Certificate` object for HTTPS responses
(:issue:`2726`, :issue:`4054`)
* A new :setting:`DNS_RESOLVER` setting allows enabling IPv6 support
(:issue:`1031`, :issue:`4227`)
* A new :setting:`SCRAPER_SLOT_MAX_ACTIVE_SIZE` setting allows configuring
the existing soft limit that pauses request downloads when the total
response data being processed is too high (:issue:`1410`, :issue:`3551`)
* A new :setting:`TWISTED_REACTOR` setting allows customizing the
:mod:`~twisted.internet.reactor` that Scrapy uses, allowing to
:doc:`enable asyncio support <topics/asyncio>` or deal with a
:ref:`common macOS issue <faq-specific-reactor>` (:issue:`2905`,
:issue:`4294`)
* Scheduler disk and memory queues may now use the class methods
``from_crawler`` or ``from_settings`` (:issue:`3884`)
* The new :attr:`Response.cb_kwargs <scrapy.http.Response.cb_kwargs>`
attribute serves as a shortcut for :attr:`Response.request.cb_kwargs
<scrapy.http.Request.cb_kwargs>` (:issue:`4331`)
* :meth:`Response.follow <scrapy.http.Response.follow>` now supports a
``flags`` parameter, for consistency with :class:`~scrapy.http.Request`
(:issue:`4277`, :issue:`4279`)
* :ref:`Item loader processors <topics-loaders-processors>` can now be
regular functions, they no longer need to be methods (:issue:`3899`)
* :class:`~scrapy.spiders.Rule` now accepts an ``errback`` parameter
(:issue:`4000`)
* :class:`~scrapy.http.Request` no longer requires a ``callback`` parameter
when an ``errback`` parameter is specified (:issue:`3586`, :issue:`4008`)
* :class:`~scrapy.logformatter.LogFormatter` now supports some additional
methods:
* :class:`~scrapy.logformatter.LogFormatter.download_error` for
download errors
* :class:`~scrapy.logformatter.LogFormatter.item_error` for exceptions
raised during item processing by :ref:`item pipelines
<topics-item-pipeline>`
* :class:`~scrapy.logformatter.LogFormatter.spider_error` for exceptions
raised from :ref:`spider callbacks <topics-spiders>`
(:issue:`374`, :issue:`3986`, :issue:`3989`, :issue:`4176`, :issue:`4188`)
* The :setting:`FEED_URI` setting now supports :class:`pathlib.Path` values
(:issue:`3731`, :issue:`4074`)
* A new :signal:`request_left_downloader` signal is sent when a request
leaves the downloader (:issue:`4303`)
* Scrapy logs a warning when it detects a request callback or errback that
uses ``yield`` but also returns a value, since the returned value would be
lost (:issue:`3484`, :issue:`3869`)
* :class:`~scrapy.spiders.Spider` objects now raise an :exc:`AttributeError`
exception if they do not have a :class:`~scrapy.spiders.Spider.start_urls`
attribute nor reimplement :class:`~scrapy.spiders.Spider.start_requests`,
but have a ``start_url`` attribute (:issue:`4133`, :issue:`4170`)
* :class:`~scrapy.exporters.BaseItemExporter` subclasses may now use
``super().__init__(**kwargs)`` instead of ``self._configure(kwargs)`` in
their ``__init__`` method, passing ``dont_fail=True`` to the parent
``__init__`` method if needed, and accessing ``kwargs`` at ``self._kwargs``
after calling their parent ``__init__`` method (:issue:`4193`,
:issue:`4370`)
* A new ``keep_fragments`` parameter of
:func:`scrapy.utils.request.request_fingerprint` allows to generate
different fingerprints for requests with different fragments in their URL
(:issue:`4104`)
* Download handlers (see :setting:`DOWNLOAD_HANDLERS`) may now use the
``from_settings`` and ``from_crawler`` class methods that other Scrapy
components already supported (:issue:`4126`)
* :class:`scrapy.utils.python.MutableChain.__iter__` now returns ``self``,
`allowing it to be used as a sequence <https://lgtm.com/rules/4850080/>`_
(:issue:`4153`)
Bug fixes
~~~~~~~~~
* The :command:`crawl` command now also exits with exit code 1 when an
exception happens before the crawling starts (:issue:`4175`, :issue:`4207`)
* :class:`LinkExtractor.extract_links
<scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor.extract_links>` no longer
re-encodes the query string or URLs from non-UTF-8 responses in UTF-8
(:issue:`998`, :issue:`1403`, :issue:`1949`, :issue:`4321`)
* The first spider middleware (see :setting:`SPIDER_MIDDLEWARES`) now also
processes exceptions raised from callbacks that are generators
(:issue:`4260`, :issue:`4272`)
* Redirects to URLs starting with 3 slashes (``///``) are now supported
(:issue:`4032`, :issue:`4042`)
* :class:`~scrapy.http.Request` no longer accepts strings as ``url`` simply
because they have a colon (:issue:`2552`, :issue:`4094`)
* The correct encoding is now used for attach names in
:class:`~scrapy.mail.MailSender` (:issue:`4229`, :issue:`4239`)
* :class:`~scrapy.dupefilters.RFPDupeFilter`, the default
:setting:`DUPEFILTER_CLASS`, no longer writes an extra ``\r`` character on
each line in Windows, which made the size of the ``requests.seen`` file
unnecessarily large on that platform (:issue:`4283`)
* Z shell auto-completion now looks for ``.html`` files, not ``.http`` files,
and covers the ``-h`` command-line switch (:issue:`4122`, :issue:`4291`)
* Adding items to a :class:`scrapy.utils.datatypes.LocalCache` object
without a ``limit`` defined no longer raises a :exc:`TypeError` exception
(:issue:`4123`)
* Fixed a typo in the message of the :exc:`ValueError` exception raised when
:func:`scrapy.utils.misc.create_instance` gets both ``settings`` and
``crawler`` set to ``None`` (:issue:`4128`)
Documentation
~~~~~~~~~~~~~
* API documentation now links to an online, syntax-highlighted view of the
corresponding source code (:issue:`4148`)
* Links to unexisting documentation pages now allow access to the sidebar
(:issue:`4152`, :issue:`4169`)
* Cross-references within our documentation now display a tooltip when
hovered (:issue:`4173`, :issue:`4183`)
* Improved the documentation about :meth:`LinkExtractor.extract_links
<scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor.extract_links>` and
simplified :ref:`topics-link-extractors` (:issue:`4045`)
* Clarified how :class:`ItemLoader.item <scrapy.loader.ItemLoader.item>`
works (:issue:`3574`, :issue:`4099`)
* Clarified that :func:`logging.basicConfig` should not be used when also
using :class:`~scrapy.crawler.CrawlerProcess` (:issue:`2149`,
:issue:`2352`, :issue:`3146`, :issue:`3960`)
* Clarified the requirements for :class:`~scrapy.http.Request` objects
:ref:`when using persistence <request-serialization>` (:issue:`4124`,
:issue:`4139`)
* Clarified how to install a :ref:`custom image pipeline
<media-pipeline-example>` (:issue:`4034`, :issue:`4252`)
* Fixed the signatures of the ``file_path`` method in :ref:`media pipeline
<topics-media-pipeline>` examples (:issue:`4290`)
* Covered a backward-incompatible change in Scrapy 1.7.0 affecting custom
:class:`scrapy.core.scheduler.Scheduler` subclasses (:issue:`4274`)
* Improved the ``README.rst`` and ``CODE_OF_CONDUCT.md`` files
(:issue:`4059`)
* Documentation examples are now checked as part of our test suite and we
have fixed some of the issues detected (:issue:`4142`, :issue:`4146`,
:issue:`4171`, :issue:`4184`, :issue:`4190`)
* Fixed logic issues, broken links and typos (:issue:`4247`, :issue:`4258`,
:issue:`4282`, :issue:`4288`, :issue:`4305`, :issue:`4308`, :issue:`4323`,
:issue:`4338`, :issue:`4359`, :issue:`4361`)
* Improved consistency when referring to the ``__init__`` method of an object
(:issue:`4086`, :issue:`4088`)
* Fixed an inconsistency between code and output in :ref:`intro-overview`
(:issue:`4213`)
* Extended :mod:`~sphinx.ext.intersphinx` usage (:issue:`4147`,
:issue:`4172`, :issue:`4185`, :issue:`4194`, :issue:`4197`)
* We now use a recent version of Python to build the documentation
(:issue:`4140`, :issue:`4249`)
* Cleaned up documentation (:issue:`4143`, :issue:`4275`)
Quality assurance
~~~~~~~~~~~~~~~~~
* Re-enabled proxy ``CONNECT`` tests (:issue:`2545`, :issue:`4114`)
* Added Bandit_ security checks to our test suite (:issue:`4162`,
:issue:`4181`)
* Added Flake8_ style checks to our test suite and applied many of the
corresponding changes (:issue:`3944`, :issue:`3945`, :issue:`4137`,
:issue:`4157`, :issue:`4167`, :issue:`4174`, :issue:`4186`, :issue:`4195`,
:issue:`4238`, :issue:`4246`, :issue:`4355`, :issue:`4360`, :issue:`4365`)
* Improved test coverage (:issue:`4097`, :issue:`4218`, :issue:`4236`)
* Started reporting slowest tests, and improved the performance of some of
them (:issue:`4163`, :issue:`4164`)
* Fixed broken tests and refactored some tests (:issue:`4014`, :issue:`4095`,
:issue:`4244`, :issue:`4268`, :issue:`4372`)
* Modified the :doc:`tox <tox:index>` configuration to allow running tests
with any Python version, run Bandit_ and Flake8_ tests by default, and
enforce a minimum tox version programmatically (:issue:`4179`)
* Cleaned up code (:issue:`3937`, :issue:`4208`, :issue:`4209`,
:issue:`4210`, :issue:`4212`, :issue:`4369`, :issue:`4376`, :issue:`4378`)
.. _Bandit: https://bandit.readthedocs.io/
.. _Flake8: https://flake8.pycqa.org/en/latest/
.. _2-0-0-scheduler-queue-changes:
Changes to scheduler queue classes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The following changes may impact any custom queue classes of all types:
* The ``push`` method no longer receives a second positional parameter
containing ``request.priority * -1``. If you need that value, get it
from the first positional parameter, ``request``, instead, or use
the new :meth:`~scrapy.core.scheduler.ScrapyPriorityQueue.priority`
method in :class:`scrapy.core.scheduler.ScrapyPriorityQueue`
subclasses.
The following changes may impact custom priority queue classes:
* In the ``__init__`` method or the ``from_crawler`` or ``from_settings``
class methods:
* The parameter that used to contain a factory function,
``qfactory``, is now passed as a keyword parameter named
``downstream_queue_cls``.
* A new keyword parameter has been added: ``key``. It is a string
that is always an empty string for memory queues and indicates the
:setting:`JOB_DIR` value for disk queues.
* The parameter for disk queues that contains data from the previous
crawl, ``startprios`` or ``slot_startprios``, is now passed as a
keyword parameter named ``startprios``.
* The ``serialize`` parameter is no longer passed. The disk queue
class must take care of request serialization on its own before
writing to disk, using the
:func:`~scrapy.utils.reqser.request_to_dict` and
:func:`~scrapy.utils.reqser.request_from_dict` functions from the
:mod:`scrapy.utils.reqser` module.
The following changes may impact custom disk and memory queue classes:
* The signature of the ``__init__`` method is now
``__init__(self, crawler, key)``.
The following changes affect specifically the
:class:`~scrapy.core.scheduler.ScrapyPriorityQueue` and
:class:`~scrapy.core.scheduler.DownloaderAwarePriorityQueue` classes from
:mod:`scrapy.core.scheduler` and may affect subclasses:
* In the ``__init__`` method, most of the changes described above apply.
``__init__`` may still receive all parameters as positional parameters,
however:
* ``downstream_queue_cls``, which replaced ``qfactory``, must be
instantiated differently.
``qfactory`` was instantiated with a priority value (integer).
Instances of ``downstream_queue_cls`` should be created using
the new
:meth:`ScrapyPriorityQueue.qfactory <scrapy.core.scheduler.ScrapyPriorityQueue.qfactory>`
or
:meth:`DownloaderAwarePriorityQueue.pqfactory <scrapy.core.scheduler.DownloaderAwarePriorityQueue.pqfactory>`
methods.
* The new ``key`` parameter displaced the ``startprios``
parameter 1 position to the right.
* The following class attributes have been added:
* :attr:`~scrapy.core.scheduler.ScrapyPriorityQueue.crawler`
* :attr:`~scrapy.core.scheduler.ScrapyPriorityQueue.downstream_queue_cls`
(details above)
* :attr:`~scrapy.core.scheduler.ScrapyPriorityQueue.key` (details above)
* The ``serialize`` attribute has been removed (details above)
The following changes affect specifically the
:class:`~scrapy.core.scheduler.ScrapyPriorityQueue` class and may affect
subclasses:
* A new :meth:`~scrapy.core.scheduler.ScrapyPriorityQueue.priority`
method has been added which, given a request, returns
``request.priority * -1``.
It is used in :meth:`~scrapy.core.scheduler.ScrapyPriorityQueue.push`
to make up for the removal of its ``priority`` parameter.
* The ``spider`` attribute has been removed. Use
:attr:`crawler.spider <scrapy.core.scheduler.ScrapyPriorityQueue.crawler>`
instead.
The following changes affect specifically the
:class:`~scrapy.core.scheduler.DownloaderAwarePriorityQueue` class and may
affect subclasses:
* A new :attr:`~scrapy.core.scheduler.DownloaderAwarePriorityQueue.pqueues`
attribute offers a mapping of downloader slot names to the
corresponding instances of
:attr:`~scrapy.core.scheduler.DownloaderAwarePriorityQueue.downstream_queue_cls`.
(:issue:`3884`)
.. _release-1.8.0:
@ -288,12 +732,12 @@ Backward-incompatible changes
:class:`~scrapy.http.Request` objects instead of arbitrary Python data
structures.
* An additional ``crawler`` parameter has been added to the ``__init__`` method
of the :class:`scrapy.core.scheduler.Scheduler` class.
Custom scheduler subclasses which don't accept arbitrary parameters in
their ``__init__`` method might break because of this change.
* An additional ``crawler`` parameter has been added to the ``__init__``
method of the :class:`~scrapy.core.scheduler.Scheduler` class. Custom
scheduler subclasses which don't accept arbitrary parameters in their
``__init__`` method might break because of this change.
For more information, refer to the documentation for the :setting:`SCHEDULER` setting.
For more information, see :setting:`SCHEDULER`.
See also :ref:`1.7-deprecation-removals` below.

28
docs/topics/asyncio.rst Normal file
View File

@ -0,0 +1,28 @@
=======
asyncio
=======
.. versionadded:: 2.0
Scrapy has partial support :mod:`asyncio`. After you :ref:`install the asyncio
reactor <install-asyncio>`, you may use :mod:`asyncio` and
:mod:`asyncio`-powered libraries in any :doc:`coroutine <coroutines>`.
.. warning:: :mod:`asyncio` support in Scrapy is experimental. Future Scrapy
versions may introduce related changes without a deprecation
period or warning.
.. _install-asyncio:
Installing the asyncio reactor
==============================
To enable :mod:`asyncio` support, set the :setting:`TWISTED_REACTOR` setting to
``'twisted.internet.asyncioreactor.AsyncioSelectorReactor'``.
If you are using :class:`~scrapy.crawler.CrawlerRunner`, you also need to
install the :class:`~twisted.internet.asyncioreactor.AsyncioSelectorReactor`
reactor manually. You can do that using
:func:`~scrapy.utils.reactor.install_reactor`::
install_reactor('twisted.internet.asyncioreactor.AsyncioSelectorReactor')

110
docs/topics/coroutines.rst Normal file
View File

@ -0,0 +1,110 @@
==========
Coroutines
==========
.. versionadded:: 2.0
Scrapy has :ref:`partial support <coroutine-support>` for the
:ref:`coroutine syntax <async>`.
.. warning:: :mod:`asyncio` support in Scrapy is experimental. Future Scrapy
versions may introduce related API and behavior changes without a
deprecation period or warning.
.. _coroutine-support:
Supported callables
===================
The following callables may be defined as coroutines using ``async def``, and
hence use coroutine syntax (e.g. ``await``, ``async for``, ``async with``):
- :class:`~scrapy.http.Request` callbacks.
The following are known caveats of the current implementation that we aim
to address in future versions of Scrapy:
- The callback output is not processed until the whole callback finishes.
As a side effect, if the callback raises an exception, none of its
output is processed.
- Because `asynchronous generators were introduced in Python 3.6`_, you
can only use ``yield`` if you are using Python 3.6 or later.
If you need to output multiple items or requests and you are using
Python 3.5, return an iterable (e.g. a list) instead.
- The :meth:`process_item` method of
:ref:`item pipelines <topics-item-pipeline>`.
- The
:meth:`~scrapy.downloadermiddlewares.DownloaderMiddleware.process_request`,
:meth:`~scrapy.downloadermiddlewares.DownloaderMiddleware.process_response`,
and
:meth:`~scrapy.downloadermiddlewares.DownloaderMiddleware.process_exception`
methods of
:ref:`downloader middlewares <topics-downloader-middleware-custom>`.
- :ref:`Signal handlers that support deferreds <signal-deferred>`.
.. _asynchronous generators were introduced in Python 3.6: https://www.python.org/dev/peps/pep-0525/
Usage
=====
There are several use cases for coroutines in Scrapy. Code that would
return Deferreds when written for previous Scrapy versions, such as downloader
middlewares and signal handlers, can be rewritten to be shorter and cleaner::
class DbPipeline:
def _update_item(self, data, item):
item['field'] = data
return item
def process_item(self, item, spider):
dfd = db.get_some_data(item['id'])
dfd.addCallback(self._update_item, item)
return dfd
becomes::
class DbPipeline:
async def process_item(self, item, spider):
item['field'] = await db.get_some_data(item['id'])
return item
Coroutines may be used to call asynchronous code. This includes other
coroutines, functions that return Deferreds and functions that return
`awaitable objects`_ such as :class:`~asyncio.Future`. This means you can use
many useful Python libraries providing such code::
class MySpider(Spider):
# ...
async def parse_with_deferred(self, response):
additional_response = await treq.get('https://additional.url')
additional_data = await treq.content(additional_response)
# ... use response and additional_data to yield items and requests
async def parse_with_asyncio(self, response):
async with aiohttp.ClientSession() as session:
async with session.get('https://additional.url') as additional_response:
additional_data = await r.text()
# ... use response and additional_data to yield items and requests
.. note:: Many libraries that use coroutines, such as `aio-libs`_, require the
:mod:`asyncio` loop and to use them you need to
:doc:`enable asyncio support in Scrapy<asyncio>`.
Common use cases for asynchronous code include:
* requesting data from websites, databases and other services (in callbacks,
pipelines and middlewares);
* storing data in databases (in pipelines and middlewares);
* delaying the spider initialization until some external event (in the
:signal:`spider_opened` handler);
* calling asynchronous Scrapy methods like ``ExecutionEngine.download`` (see
:ref:`the screenshot pipeline example<ScreenshotPipeline>`).
.. _aio-libs: https://github.com/aio-libs
.. _awaitable objects: https://docs.python.org/3/glossary.html#term-awaitable

4
docs/topics/downloader-middleware.rst
View File

@ -872,6 +872,10 @@ Default: ``[]``
Meta tags within these tags are ignored.
.. versionchanged:: 2.0
The default value of :setting:`METAREFRESH_IGNORE_TAGS` changed from
``['script', 'noscript']`` to ``[]``.
.. setting:: METAREFRESH_MAXDELAY
METAREFRESH_MAXDELAY

5
docs/topics/exporters.rst
View File

@ -137,7 +137,7 @@ output examples, which assume you're exporting these two items::
BaseItemExporter
----------------
.. class:: BaseItemExporter(fields_to_export=None, export_empty_fields=False, encoding='utf-8', indent=0)
.. class:: BaseItemExporter(fields_to_export=None, export_empty_fields=False, encoding='utf-8', indent=0, dont_fail=False)
This is the (abstract) base class for all Item Exporters. It provides
support for common features used by all (concrete) Item Exporters, such as
@ -148,6 +148,9 @@ BaseItemExporter
populate their respective instance attributes: :attr:`fields_to_export`,
:attr:`export_empty_fields`, :attr:`encoding`, :attr:`indent`.
.. versionadded:: 2.0
The *dont_fail* parameter.
.. method:: export_item(item)
Exports the given item. This method must be implemented in subclasses.

3
docs/topics/feed-exports.rst
View File

@ -236,6 +236,9 @@ supported URI schemes.
This setting is required for enabling the feed exports.
.. versionchanged:: 2.0
Added :class:`pathlib.Path` support.
.. setting:: FEED_FORMAT
FEED_FORMAT

13
docs/topics/item-pipeline.rst
View File

@ -162,14 +162,16 @@ method and how to clean up the resources properly.::
.. _pymongo: https://api.mongodb.com/python/current/
.. _ScreenshotPipeline:
Take screenshot of item
-----------------------
This example demonstrates how to return a
:class:`~twisted.internet.defer.Deferred` from the :meth:`process_item` method.
It uses Splash_ to render screenshot of item url. Pipeline
makes request to locally running instance of Splash_. After request is downloaded
and Deferred callback fires, it saves item to a file and adds filename to an item.
makes request to locally running instance of Splash_. After request is downloaded,
it saves the screenshot to a file and adds filename to the item.
::
@ -184,15 +186,12 @@ and Deferred callback fires, it saves item to a file and adds filename to an ite
SPLASH_URL = "http://localhost:8050/render.png?url={}"
def process_item(self, item, spider):
async def process_item(self, item, spider):
encoded_item_url = quote(item["url"])
screenshot_url = self.SPLASH_URL.format(encoded_item_url)
request = scrapy.Request(screenshot_url)
dfd = spider.crawler.engine.download(request, spider)
dfd.addBoth(self.return_item, item)
return dfd
response = await spider.crawler.engine.download(request, spider)
def return_item(self, response, item):
if response.status != 200:
# Error happened, return item.
return item

3
docs/topics/jobs.rst
View File

@ -68,6 +68,9 @@ Cookies may expire. So, if you don't resume your spider quickly the requests
scheduled may no longer work. This won't be an issue if you spider doesn't rely
on cookies.
.. _request-serialization:
Request serialization
---------------------

10
docs/topics/link-extractors.rst
View File

@ -64,9 +64,13 @@ LxmlLinkExtractor
:param deny_extensions: a single value or list of strings containing
extensions that should be ignored when extracting links.
If not given, it will default to the
``IGNORED_EXTENSIONS`` list defined in the
`scrapy.linkextractors`_ package.
If not given, it will default to
:data:`scrapy.linkextractors.IGNORED_EXTENSIONS`.
.. versionchanged:: 2.0
:data:`~scrapy.linkextractors.IGNORED_EXTENSIONS` now includes
``7z``, ``7zip``, ``apk``, ``bz2``, ``cdr``, ``dmg``, ``ico``,
``iso``, ``tar``, ``tar.gz``, ``webm``, and ``xz``.
:type deny_extensions: list
:param restrict_xpaths: is an XPath (or list of XPath's) which defines

3
docs/topics/loaders.rst
View File

@ -136,6 +136,9 @@ with the data to be parsed, and return a parsed value. So you can use any
function as input or output processor. The only requirement is that they must
accept one (and only one) positional argument, which will be an iterable.
.. versionchanged:: 2.0
Processors no longer need to be methods.
.. note:: Both input and output processors must receive an iterable as their
first argument. The output of those functions can be anything. The result of
input processors will be appended to an internal list (in the Loader)

12
docs/topics/media-pipeline.rst
View File

@ -116,12 +116,6 @@ For the Images Pipeline, set the :setting:`IMAGES_STORE` setting::
Supported Storage
=================
File system is currently the only officially supported storage, but there are
also support for storing files in `Amazon S3`_ and `Google Cloud Storage`_.
.. _Amazon S3: https://aws.amazon.com/s3/
.. _Google Cloud Storage: https://cloud.google.com/storage/
File system storage
-------------------
@ -147,9 +141,13 @@ Where:
* ``full`` is a sub-directory to separate full images from thumbnails (if
used). For more info see :ref:`topics-images-thumbnails`.
.. _media-pipeline-ftp:
FTP server storage
------------------
.. versionadded:: 2.0
:setting:`FILES_STORE` and :setting:`IMAGES_STORE` can point to an FTP server.
Scrapy will automatically upload the files to the server.
@ -573,6 +571,8 @@ See here the methods that you can override in your custom Images Pipeline:
By default, the :meth:`item_completed` method returns the item.
.. _media-pipeline-example:
Custom Images pipeline example
==============================

8
docs/topics/request-response.rst
View File

@ -31,6 +31,8 @@ Request objects
a :class:`Response`.
:param url: the URL of this request
If the URL is invalid, a :exc:`ValueError` exception is raised.
:type url: string
:param callback: the function that will be called with the response of this
@ -125,6 +127,10 @@ Request objects
:exc:`~twisted.python.failure.Failure` as first parameter.
For more information,
see :ref:`topics-request-response-ref-errbacks` below.
.. versionchanged:: 2.0
The *callback* parameter is no longer required when the *errback*
parameter is specified.
:type errback: callable
:param flags: Flags sent to the request, can be used for logging or similar purposes.
@ -677,6 +683,8 @@ Response objects
.. attribute:: Response.cb_kwargs
.. versionadded:: 2.0
A shortcut to the :attr:`Request.cb_kwargs` attribute of the
:attr:`Response.request` object (i.e. ``self.request.cb_kwargs``).

43
docs/topics/settings.rst
View File

@ -381,6 +381,8 @@ DNS in-memory cache size.
DNS_RESOLVER
------------
.. versionadded:: 2.0
Default: ``'scrapy.resolver.CachingThreadedResolver'``
The class to be used to resolve DNS names. The default ``scrapy.resolver.CachingThreadedResolver``
@ -1258,6 +1260,9 @@ does not work together with :setting:`CONCURRENT_REQUESTS_PER_IP`.
SCRAPER_SLOT_MAX_ACTIVE_SIZE
----------------------------
.. versionadded:: 2.0
Default: ``5_000_000``
Soft limit (in bytes) for response data being processed.
@ -1447,24 +1452,36 @@ in the ``project`` subdirectory.
TWISTED_REACTOR
---------------
.. versionadded:: 2.0
Default: ``None``
Import path of a given Twisted reactor, for instance:
:class:`twisted.internet.asyncioreactor.AsyncioSelectorReactor`.
Import path of a given :mod:`~twisted.internet.reactor`.
Scrapy will install this reactor if no other is installed yet, such as when
the ``scrapy`` CLI program is invoked or when using the
:class:`~scrapy.crawler.CrawlerProcess` class. If you are using the
:class:`~scrapy.crawler.CrawlerRunner` class, you need to install the correct
reactor manually. An exception will be raised if the installation fails.
Scrapy will install this reactor if no other reactor is installed yet, such as
when the ``scrapy`` CLI program is invoked or when using the
:class:`~scrapy.crawler.CrawlerProcess` class.
The default value for this option is currently ``None``, which means that Scrapy
will not attempt to install any specific reactor, and the default one defined by
Twisted for the current platform will be used. This is to maintain backward
compatibility and avoid possible problems caused by using a non-default reactor.
If you are using the :class:`~scrapy.crawler.CrawlerRunner` class, you also
need to install the correct reactor manually. You can do that using
:func:`~scrapy.utils.reactor.install_reactor`:
For additional information, please see
:doc:`core/howto/choosing-reactor`.
.. autofunction:: scrapy.utils.reactor.install_reactor
If a reactor is already installed,
:func:`~scrapy.utils.reactor.install_reactor` has no effect.
:meth:`CrawlerRunner.__init__ <scrapy.crawler.CrawlerRunner.__init__>` raises
:exc:`Exception` if the installed reactor does not match the
:setting:`TWISTED_REACTOR` setting.
The default value of the :setting:`TWISTED_REACTOR` setting is ``None``, which
means that Scrapy will not attempt to install any specific reactor, and the
default reactor defined by Twisted for the current platform will be used. This
is to maintain backward compatibility and avoid possible problems caused by
using a non-default reactor.
For additional information, see :doc:`core/howto/choosing-reactor`.
.. setting:: URLLENGTH_LIMIT

3
docs/topics/signals.rst
View File

@ -46,6 +46,7 @@ Here is a simple example showing how you can catch signals and perform some acti
def parse(self, response):
pass
.. _signal-deferred:
Deferred signal handlers
========================
@ -301,6 +302,8 @@ request_left_downloader
.. signal:: request_left_downloader
.. function:: request_left_downloader(request, spider)
.. versionadded:: 2.0
Sent when a :class:`~scrapy.http.Request` leaves the downloader, even in case of
failure.

3
docs/topics/spiders.rst
View File

@ -420,6 +420,9 @@ Crawling rules
It receives a :class:`Twisted Failure <twisted.python.failure.Failure>`
instance as first parameter.
.. versionadded:: 2.0
The *errback* parameter.
CrawlSpider example
~~~~~~~~~~~~~~~~~~~

5
scrapy/http/response/__init__.py
View File

@ -129,6 +129,9 @@ class Response(object_ref):
:class:`~.TextResponse` provides a :meth:`~.TextResponse.follow`
method which supports selectors in addition to absolute/relative URLs
and Link objects.
.. versionadded:: 2.0
The *flags* parameter.
"""
if isinstance(url, Link):
url = url.url
@ -157,6 +160,8 @@ class Response(object_ref):
dont_filter=False, errback=None, cb_kwargs=None, flags=None):
# type: (...) -> Generator[Request, None, None]
"""
.. versionadded:: 2.0
Return an iterable of :class:`~.Request` instances to follow all links
in ``urls``. It accepts the same arguments as ``Request.__init__`` method,
but elements of ``urls`` can be relative URLs or :class:`~scrapy.link.Link` objects,

17
scrapy/logformatter.py
View File

@ -97,7 +97,11 @@ class LogFormatter(object):
}
def item_error(self, item, exception, response, spider):
"""Logs a message when an item causes an error while it is passing through the item pipeline."""
"""Logs a message when an item causes an error while it is passing
through the item pipeline.
.. versionadded:: 2.0
"""
return {
'level': logging.ERROR,
'msg': ITEMERRORMSG,
@ -107,7 +111,10 @@ class LogFormatter(object):
}
def spider_error(self, failure, request, response, spider):
"""Logs an error message from a spider."""
"""Logs an error message from a spider.
.. versionadded:: 2.0
"""
return {
'level': logging.ERROR,
'msg': SPIDERERRORMSG,
@ -118,7 +125,11 @@ class LogFormatter(object):
}
def download_error(self, failure, request, spider, errmsg=None):
"""Logs a download error message from a spider (typically coming from the engine)."""
"""Logs a download error message from a spider (typically coming from
the engine).
.. versionadded:: 2.0
"""
args = {'request': request}
if errmsg:
msg = DOWNLOADERRORMSG_LONG

5
scrapy/utils/reactor.py
View File

@ -50,6 +50,8 @@ class CallLaterOnce(object):
def install_reactor(reactor_path):
"""Installs the :mod:`~twisted.internet.reactor` with the specified
import path."""
reactor_class = load_object(reactor_path)
if reactor_class is asyncioreactor.AsyncioSelectorReactor:
with suppress(error.ReactorAlreadyInstalledError):
@ -63,6 +65,9 @@ def install_reactor(reactor_path):
def verify_installed_reactor(reactor_path):
"""Raises :exc:`Exception` if the installed
:mod:`~twisted.internet.reactor` does not match the specified import
path."""
from twisted.internet import reactor
reactor_class = load_object(reactor_path)
if not isinstance(reactor, reactor_class):

2
tests/test_crawl.py
View File

@ -325,7 +325,7 @@ with multiples lines
@mark.only_asyncio()
@defer.inlineCallbacks
def test_async_def_asyncio_parse(self):
runner = CrawlerRunner({"ASYNCIO_REACTOR": True})
runner = CrawlerRunner({"TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor"})
runner.crawl(AsyncDefAsyncioSpider, self.mockserver.url("/status?n=200"), mockserver=self.mockserver)
with LogCapture() as log:
yield runner.join()