diff --git a/docs/intro/overview.rst b/docs/intro/overview.rst index 488e1c2e3..5937c5860 100644 --- a/docs/intro/overview.rst +++ b/docs/intro/overview.rst @@ -83,7 +83,7 @@ While this enables you to do very fast crawls (sending multiple concurrent requests at the same time, in a fault-tolerant way) Scrapy also gives you control over the politeness of the crawl through :ref:`a few settings `. You can do things like setting a download delay between -each request, limiting the amount of concurrent requests per domain or per IP, and +each request, limiting the amount of concurrent requests per domain, and even :ref:`using an auto-throttling extension ` that tries to figure these settings out automatically. diff --git a/docs/topics/asyncio.rst b/docs/topics/asyncio.rst index 17b322b5a..3d4ebe088 100644 --- a/docs/topics/asyncio.rst +++ b/docs/topics/asyncio.rst @@ -5,7 +5,7 @@ asyncio ======= Scrapy supports :mod:`asyncio` natively. New projects created with -:command:`scrapy startproject` have asyncio enabled by default, and you can use +:command:`startproject` have asyncio enabled by default, and you can use :mod:`asyncio` and :mod:`asyncio`-powered libraries in any :doc:`coroutine `. @@ -18,7 +18,7 @@ no additional setup is needed. Configuring the asyncio reactor =============================== -New projects generated with :command:`scrapy startproject` have the asyncio +New projects generated with :command:`startproject` have the asyncio reactor configured by default. No manual setup is needed. The :setting:`TWISTED_REACTOR` setting controls which Twisted reactor Scrapy diff --git a/docs/topics/coroutines.rst b/docs/topics/coroutines.rst index 8aaf0aa31..3af4b2fad 100644 --- a/docs/topics/coroutines.rst +++ b/docs/topics/coroutines.rst @@ -16,7 +16,7 @@ Supported callables The following callables may be defined as coroutines using ``async def``, and hence use coroutine syntax (e.g. ``await``, ``async for``, ``async with``): -- The :meth:`~scrapy.spiders.Spider.start` spider method, which *must* be +- The :meth:`~scrapy.Spider.start` spider method, which *must* be defined as an :term:`asynchronous generator`. .. versionadded:: 2.13 @@ -204,13 +204,14 @@ This means you can use many useful Python libraries providing such code: Common use cases for asynchronous code include: * requesting data from websites, databases and other services (in - :meth:`~scrapy.spiders.Spider.start`, callbacks, pipelines and + :meth:`~scrapy.Spider.start`, callbacks, pipelines and middlewares); * storing data in databases (in pipelines and middlewares); * delaying the spider initialization until some external event (in the :signal:`spider_opened` handler); -* calling asynchronous Scrapy methods like :meth:`ExecutionEngine.download` - (see :ref:`the screenshot pipeline example`). +* calling asynchronous Scrapy methods like + :meth:`scrapy.core.engine.ExecutionEngine.download_async` (see + :ref:`the screenshot pipeline example`). .. _aio-libs: https://github.com/aio-libs diff --git a/docs/topics/download-handlers.rst b/docs/topics/download-handlers.rst index 9bbfee9e2..5fbc5df06 100644 --- a/docs/topics/download-handlers.rst +++ b/docs/topics/download-handlers.rst @@ -89,7 +89,7 @@ the following API: If ``True``, the handler will only be instantiated when the first request handled by it needs to be downloaded. - .. method:: download_request(request: Request) -> Response: + .. method:: download_request(request: Request) -> Response :async: Download the given request and return a response. diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst index 64ff02690..0a873afd6 100644 --- a/docs/topics/downloader-middleware.rst +++ b/docs/topics/downloader-middleware.rst @@ -591,8 +591,8 @@ In order to use your storage backend, set: HTTPCache middleware settings ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -The :class:`HttpCacheMiddleware` can be configured through the following -settings: +The :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware` can be +configured through the following settings: .. setting:: HTTPCACHE_ENABLED @@ -868,9 +868,9 @@ OffsiteMiddleware .. reqmeta:: allow_offsite If the request has the :attr:`~scrapy.Request.dont_filter` attribute set to - ``True`` or :attr:`Request.meta` has ``allow_offsite`` set to ``True``, then - the OffsiteMiddleware will allow the request even if its domain is not listed - in allowed domains. + ``True`` or :attr:`Request.meta ` has ``allow_offsite`` + set to ``True``, then the OffsiteMiddleware will allow the request even if + its domain is not listed in allowed domains. RedirectMiddleware ------------------ diff --git a/docs/topics/items.rst b/docs/topics/items.rst index a8c4b33a7..0892dd839 100644 --- a/docs/topics/items.rst +++ b/docs/topics/items.rst @@ -62,8 +62,8 @@ its ``__init__`` method. :class:`Item` also allows the defining of field metadata, which can be used to :ref:`customize serialization `. -:mod:`trackref` tracks :class:`Item` objects to help find memory leaks -(see :ref:`topics-leaks-trackrefs`). +:mod:`scrapy.utils.trackref` tracks :class:`Item` objects to help find memory +leaks (see :ref:`topics-leaks-trackrefs`). Example: diff --git a/docs/topics/leaks.rst b/docs/topics/leaks.rst index 70d563fdd..3337cc5d0 100644 --- a/docs/topics/leaks.rst +++ b/docs/topics/leaks.rst @@ -62,9 +62,9 @@ Debugging memory leaks with ``trackref`` .. skip: start -:mod:`trackref` is a module provided by Scrapy to debug the most common cases of -memory leaks. It basically tracks the references to all live Request, -Response, Item, Spider and Selector objects. +:mod:`scrapy.utils.trackref` is a module provided by Scrapy to debug the most +common cases of memory leaks. It basically tracks the references to all live +Request, Response, Item, Spider and Selector objects. You can enter the telnet console and inspect how many objects (of the classes mentioned above) are currently alive using the ``prefs()`` function which is an @@ -164,7 +164,7 @@ Too many spiders? ----------------- If your project has too many spiders executed in parallel, -the output of :func:`prefs` can be difficult to read. +the output of ``prefs()`` can be difficult to read. For this reason, that function has a ``ignore`` argument which can be used to ignore a particular class (and all its subclasses). For example, this won't show any live references to spiders: diff --git a/scrapy/core/scheduler.py b/scrapy/core/scheduler.py index 82e31b90b..42c517222 100644 --- a/scrapy/core/scheduler.py +++ b/scrapy/core/scheduler.py @@ -289,11 +289,11 @@ class Scheduler(BaseScheduler): :param dqclass: A class to be used as persistent request queue. The value for the :setting:`SCHEDULER_DISK_QUEUE` setting is used by default. - :type dqclass: class + :type dqclass: type :param mqclass: A class to be used as non-persistent request queue. The value for the :setting:`SCHEDULER_MEMORY_QUEUE` setting is used by default. - :type mqclass: class + :type mqclass: type :param logunser: A boolean that indicates whether or not unserializable requests should be logged. The value for the :setting:`SCHEDULER_DEBUG` setting is used by default. @@ -306,7 +306,7 @@ class Scheduler(BaseScheduler): :param pqclass: A class to be used as priority queue for requests. The value for the :setting:`SCHEDULER_PRIORITY_QUEUE` setting is used by default. - :type pqclass: class + :type pqclass: type :param crawler: The crawler object corresponding to the current crawl. :type crawler: :class:`scrapy.crawler.Crawler` diff --git a/scrapy/settings/__init__.py b/scrapy/settings/__init__.py index 932463ba9..27b0f1187 100644 --- a/scrapy/settings/__init__.py +++ b/scrapy/settings/__init__.py @@ -288,10 +288,10 @@ class BaseSettings(MutableMapping[str, Any]): - ``['one', 'two']`` if set to ``'["one", "two"]'`` or ``'one,two'`` :param name: the setting name - :type name: string + :type name: str :param default: the value to return if no setting is found - :type default: any + :type default: object """ value = self.get(name, default) if value is None: