mirror of https://github.com/scrapy/scrapy.git
Second pass.
This commit is contained in:
parent
a328260e2c
commit
51410512b7
|
|
@ -83,7 +83,7 @@ While this enables you to do very fast crawls (sending multiple concurrent
|
|||
requests at the same time, in a fault-tolerant way) Scrapy also gives you
|
||||
control over the politeness of the crawl through :ref:`a few settings
|
||||
<topics-settings-ref>`. You can do things like setting a download delay between
|
||||
each request, limiting the amount of concurrent requests per domain or per IP, and
|
||||
each request, limiting the amount of concurrent requests per domain, and
|
||||
even :ref:`using an auto-throttling extension <topics-autothrottle>` that tries
|
||||
to figure these settings out automatically.
|
||||
|
||||
|
|
|
|||
|
|
@ -5,7 +5,7 @@ asyncio
|
|||
=======
|
||||
|
||||
Scrapy supports :mod:`asyncio` natively. New projects created with
|
||||
:command:`scrapy startproject` have asyncio enabled by default, and you can use
|
||||
:command:`startproject` have asyncio enabled by default, and you can use
|
||||
:mod:`asyncio` and :mod:`asyncio`-powered libraries in any :doc:`coroutine
|
||||
<coroutines>`.
|
||||
|
||||
|
|
@ -18,7 +18,7 @@ no additional setup is needed.
|
|||
Configuring the asyncio reactor
|
||||
===============================
|
||||
|
||||
New projects generated with :command:`scrapy startproject` have the asyncio
|
||||
New projects generated with :command:`startproject` have the asyncio
|
||||
reactor configured by default. No manual setup is needed.
|
||||
|
||||
The :setting:`TWISTED_REACTOR` setting controls which Twisted reactor Scrapy
|
||||
|
|
|
|||
|
|
@ -16,7 +16,7 @@ Supported callables
|
|||
The following callables may be defined as coroutines using ``async def``, and
|
||||
hence use coroutine syntax (e.g. ``await``, ``async for``, ``async with``):
|
||||
|
||||
- The :meth:`~scrapy.spiders.Spider.start` spider method, which *must* be
|
||||
- The :meth:`~scrapy.Spider.start` spider method, which *must* be
|
||||
defined as an :term:`asynchronous generator`.
|
||||
|
||||
.. versionadded:: 2.13
|
||||
|
|
@ -204,13 +204,14 @@ This means you can use many useful Python libraries providing such code:
|
|||
Common use cases for asynchronous code include:
|
||||
|
||||
* requesting data from websites, databases and other services (in
|
||||
:meth:`~scrapy.spiders.Spider.start`, callbacks, pipelines and
|
||||
:meth:`~scrapy.Spider.start`, callbacks, pipelines and
|
||||
middlewares);
|
||||
* storing data in databases (in pipelines and middlewares);
|
||||
* delaying the spider initialization until some external event (in the
|
||||
:signal:`spider_opened` handler);
|
||||
* calling asynchronous Scrapy methods like :meth:`ExecutionEngine.download`
|
||||
(see :ref:`the screenshot pipeline example<ScreenshotPipeline>`).
|
||||
* calling asynchronous Scrapy methods like
|
||||
:meth:`scrapy.core.engine.ExecutionEngine.download_async` (see
|
||||
:ref:`the screenshot pipeline example<ScreenshotPipeline>`).
|
||||
|
||||
.. _aio-libs: https://github.com/aio-libs
|
||||
|
||||
|
|
|
|||
|
|
@ -89,7 +89,7 @@ the following API:
|
|||
If ``True``, the handler will only be instantiated when the first
|
||||
request handled by it needs to be downloaded.
|
||||
|
||||
.. method:: download_request(request: Request) -> Response:
|
||||
.. method:: download_request(request: Request) -> Response
|
||||
:async:
|
||||
|
||||
Download the given request and return a response.
|
||||
|
|
|
|||
|
|
@ -591,8 +591,8 @@ In order to use your storage backend, set:
|
|||
HTTPCache middleware settings
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The :class:`HttpCacheMiddleware` can be configured through the following
|
||||
settings:
|
||||
The :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware` can be
|
||||
configured through the following settings:
|
||||
|
||||
.. setting:: HTTPCACHE_ENABLED
|
||||
|
||||
|
|
@ -868,9 +868,9 @@ OffsiteMiddleware
|
|||
.. reqmeta:: allow_offsite
|
||||
|
||||
If the request has the :attr:`~scrapy.Request.dont_filter` attribute set to
|
||||
``True`` or :attr:`Request.meta` has ``allow_offsite`` set to ``True``, then
|
||||
the OffsiteMiddleware will allow the request even if its domain is not listed
|
||||
in allowed domains.
|
||||
``True`` or :attr:`Request.meta <scrapy.Request.meta>` has ``allow_offsite``
|
||||
set to ``True``, then the OffsiteMiddleware will allow the request even if
|
||||
its domain is not listed in allowed domains.
|
||||
|
||||
RedirectMiddleware
|
||||
------------------
|
||||
|
|
|
|||
|
|
@ -62,8 +62,8 @@ its ``__init__`` method.
|
|||
:class:`Item` also allows the defining of field metadata, which can be used to
|
||||
:ref:`customize serialization <topics-exporters-field-serialization>`.
|
||||
|
||||
:mod:`trackref` tracks :class:`Item` objects to help find memory leaks
|
||||
(see :ref:`topics-leaks-trackrefs`).
|
||||
:mod:`scrapy.utils.trackref` tracks :class:`Item` objects to help find memory
|
||||
leaks (see :ref:`topics-leaks-trackrefs`).
|
||||
|
||||
Example:
|
||||
|
||||
|
|
|
|||
|
|
@ -62,9 +62,9 @@ Debugging memory leaks with ``trackref``
|
|||
|
||||
.. skip: start
|
||||
|
||||
:mod:`trackref` is a module provided by Scrapy to debug the most common cases of
|
||||
memory leaks. It basically tracks the references to all live Request,
|
||||
Response, Item, Spider and Selector objects.
|
||||
:mod:`scrapy.utils.trackref` is a module provided by Scrapy to debug the most
|
||||
common cases of memory leaks. It basically tracks the references to all live
|
||||
Request, Response, Item, Spider and Selector objects.
|
||||
|
||||
You can enter the telnet console and inspect how many objects (of the classes
|
||||
mentioned above) are currently alive using the ``prefs()`` function which is an
|
||||
|
|
@ -164,7 +164,7 @@ Too many spiders?
|
|||
-----------------
|
||||
|
||||
If your project has too many spiders executed in parallel,
|
||||
the output of :func:`prefs` can be difficult to read.
|
||||
the output of ``prefs()`` can be difficult to read.
|
||||
For this reason, that function has a ``ignore`` argument which can be used to
|
||||
ignore a particular class (and all its subclasses). For
|
||||
example, this won't show any live references to spiders:
|
||||
|
|
|
|||
|
|
@ -289,11 +289,11 @@ class Scheduler(BaseScheduler):
|
|||
|
||||
:param dqclass: A class to be used as persistent request queue.
|
||||
The value for the :setting:`SCHEDULER_DISK_QUEUE` setting is used by default.
|
||||
:type dqclass: class
|
||||
:type dqclass: type
|
||||
|
||||
:param mqclass: A class to be used as non-persistent request queue.
|
||||
The value for the :setting:`SCHEDULER_MEMORY_QUEUE` setting is used by default.
|
||||
:type mqclass: class
|
||||
:type mqclass: type
|
||||
|
||||
:param logunser: A boolean that indicates whether or not unserializable requests should be logged.
|
||||
The value for the :setting:`SCHEDULER_DEBUG` setting is used by default.
|
||||
|
|
@ -306,7 +306,7 @@ class Scheduler(BaseScheduler):
|
|||
|
||||
:param pqclass: A class to be used as priority queue for requests.
|
||||
The value for the :setting:`SCHEDULER_PRIORITY_QUEUE` setting is used by default.
|
||||
:type pqclass: class
|
||||
:type pqclass: type
|
||||
|
||||
:param crawler: The crawler object corresponding to the current crawl.
|
||||
:type crawler: :class:`scrapy.crawler.Crawler`
|
||||
|
|
|
|||
|
|
@ -288,10 +288,10 @@ class BaseSettings(MutableMapping[str, Any]):
|
|||
- ``['one', 'two']`` if set to ``'["one", "two"]'`` or ``'one,two'``
|
||||
|
||||
:param name: the setting name
|
||||
:type name: string
|
||||
:type name: str
|
||||
|
||||
:param default: the value to return if no setting is found
|
||||
:type default: any
|
||||
:type default: object
|
||||
"""
|
||||
value = self.get(name, default)
|
||||
if value is None:
|
||||
|
|
|
|||
Loading…
Reference in New Issue