Second pass.

This commit is contained in:
Andrey Rakhmatullin 2026-07-08 12:54:05 +05:00
parent a328260e2c
commit 51410512b7
9 changed files with 25 additions and 24 deletions

View File

@ -83,7 +83,7 @@ While this enables you to do very fast crawls (sending multiple concurrent
requests at the same time, in a fault-tolerant way) Scrapy also gives you
control over the politeness of the crawl through :ref:`a few settings
<topics-settings-ref>`. You can do things like setting a download delay between
each request, limiting the amount of concurrent requests per domain or per IP, and
each request, limiting the amount of concurrent requests per domain, and
even :ref:`using an auto-throttling extension <topics-autothrottle>` that tries
to figure these settings out automatically.

View File

@ -5,7 +5,7 @@ asyncio
=======
Scrapy supports :mod:`asyncio` natively. New projects created with
:command:`scrapy startproject` have asyncio enabled by default, and you can use
:command:`startproject` have asyncio enabled by default, and you can use
:mod:`asyncio` and :mod:`asyncio`-powered libraries in any :doc:`coroutine
<coroutines>`.
@ -18,7 +18,7 @@ no additional setup is needed.
Configuring the asyncio reactor
===============================
New projects generated with :command:`scrapy startproject` have the asyncio
New projects generated with :command:`startproject` have the asyncio
reactor configured by default. No manual setup is needed.
The :setting:`TWISTED_REACTOR` setting controls which Twisted reactor Scrapy

View File

@ -16,7 +16,7 @@ Supported callables
The following callables may be defined as coroutines using ``async def``, and
hence use coroutine syntax (e.g. ``await``, ``async for``, ``async with``):
- The :meth:`~scrapy.spiders.Spider.start` spider method, which *must* be
- The :meth:`~scrapy.Spider.start` spider method, which *must* be
defined as an :term:`asynchronous generator`.
.. versionadded:: 2.13
@ -204,13 +204,14 @@ This means you can use many useful Python libraries providing such code:
Common use cases for asynchronous code include:
* requesting data from websites, databases and other services (in
:meth:`~scrapy.spiders.Spider.start`, callbacks, pipelines and
:meth:`~scrapy.Spider.start`, callbacks, pipelines and
middlewares);
* storing data in databases (in pipelines and middlewares);
* delaying the spider initialization until some external event (in the
:signal:`spider_opened` handler);
* calling asynchronous Scrapy methods like :meth:`ExecutionEngine.download`
(see :ref:`the screenshot pipeline example<ScreenshotPipeline>`).
* calling asynchronous Scrapy methods like
:meth:`scrapy.core.engine.ExecutionEngine.download_async` (see
:ref:`the screenshot pipeline example<ScreenshotPipeline>`).
.. _aio-libs: https://github.com/aio-libs

View File

@ -89,7 +89,7 @@ the following API:
If ``True``, the handler will only be instantiated when the first
request handled by it needs to be downloaded.
.. method:: download_request(request: Request) -> Response:
.. method:: download_request(request: Request) -> Response
:async:
Download the given request and return a response.

View File

@ -591,8 +591,8 @@ In order to use your storage backend, set:
HTTPCache middleware settings
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The :class:`HttpCacheMiddleware` can be configured through the following
settings:
The :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware` can be
configured through the following settings:
.. setting:: HTTPCACHE_ENABLED
@ -868,9 +868,9 @@ OffsiteMiddleware
.. reqmeta:: allow_offsite
If the request has the :attr:`~scrapy.Request.dont_filter` attribute set to
``True`` or :attr:`Request.meta` has ``allow_offsite`` set to ``True``, then
the OffsiteMiddleware will allow the request even if its domain is not listed
in allowed domains.
``True`` or :attr:`Request.meta <scrapy.Request.meta>` has ``allow_offsite``
set to ``True``, then the OffsiteMiddleware will allow the request even if
its domain is not listed in allowed domains.
RedirectMiddleware
------------------

View File

@ -62,8 +62,8 @@ its ``__init__`` method.
:class:`Item` also allows the defining of field metadata, which can be used to
:ref:`customize serialization <topics-exporters-field-serialization>`.
:mod:`trackref` tracks :class:`Item` objects to help find memory leaks
(see :ref:`topics-leaks-trackrefs`).
:mod:`scrapy.utils.trackref` tracks :class:`Item` objects to help find memory
leaks (see :ref:`topics-leaks-trackrefs`).
Example:

View File

@ -62,9 +62,9 @@ Debugging memory leaks with ``trackref``
.. skip: start
:mod:`trackref` is a module provided by Scrapy to debug the most common cases of
memory leaks. It basically tracks the references to all live Request,
Response, Item, Spider and Selector objects.
:mod:`scrapy.utils.trackref` is a module provided by Scrapy to debug the most
common cases of memory leaks. It basically tracks the references to all live
Request, Response, Item, Spider and Selector objects.
You can enter the telnet console and inspect how many objects (of the classes
mentioned above) are currently alive using the ``prefs()`` function which is an
@ -164,7 +164,7 @@ Too many spiders?
-----------------
If your project has too many spiders executed in parallel,
the output of :func:`prefs` can be difficult to read.
the output of ``prefs()`` can be difficult to read.
For this reason, that function has a ``ignore`` argument which can be used to
ignore a particular class (and all its subclasses). For
example, this won't show any live references to spiders:

View File

@ -289,11 +289,11 @@ class Scheduler(BaseScheduler):
:param dqclass: A class to be used as persistent request queue.
The value for the :setting:`SCHEDULER_DISK_QUEUE` setting is used by default.
:type dqclass: class
:type dqclass: type
:param mqclass: A class to be used as non-persistent request queue.
The value for the :setting:`SCHEDULER_MEMORY_QUEUE` setting is used by default.
:type mqclass: class
:type mqclass: type
:param logunser: A boolean that indicates whether or not unserializable requests should be logged.
The value for the :setting:`SCHEDULER_DEBUG` setting is used by default.
@ -306,7 +306,7 @@ class Scheduler(BaseScheduler):
:param pqclass: A class to be used as priority queue for requests.
The value for the :setting:`SCHEDULER_PRIORITY_QUEUE` setting is used by default.
:type pqclass: class
:type pqclass: type
:param crawler: The crawler object corresponding to the current crawl.
:type crawler: :class:`scrapy.crawler.Crawler`

View File

@ -288,10 +288,10 @@ class BaseSettings(MutableMapping[str, Any]):
- ``['one', 'two']`` if set to ``'["one", "two"]'`` or ``'one,two'``
:param name: the setting name
:type name: string
:type name: str
:param default: the value to return if no setting is found
:type default: any
:type default: object
"""
value = self.get(name, default)
if value is None: