diff --git a/docs/faq.rst b/docs/faq.rst index a396a8212..8f2013581 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -384,8 +384,9 @@ How to deal with ``: filedescriptor out of range in select() ---------------------------------------------------------------------------------------------- This issue `has been reported`_ to appear when running broad crawls in macOS, where the default -Twisted reactor is :class:`twisted.internet.selectreactor.SelectReactor`. Switching to a -different reactor is possible by using the :setting:`TWISTED_REACTOR` setting. +Twisted reactor was :class:`twisted.internet.selectreactor.SelectReactor` at that time. +If you have switched to this reactor using the :setting:`TWISTED_REACTOR` setting you can switch +to a different one in the same way. .. _faq-stop-response-download: diff --git a/docs/intro/install.rst b/docs/intro/install.rst index 8cef04ff1..67a84e538 100644 --- a/docs/intro/install.rst +++ b/docs/intro/install.rst @@ -230,8 +230,8 @@ Installing Scrapy with PyPy on Windows is not tested. You can check that Scrapy is installed correctly by running ``scrapy bench``. If this command gives errors such as ``TypeError: ... got 2 unexpected keyword arguments``, this means -that setuptools was unable to pick up one PyPy-specific dependency. -To fix this issue, run ``pip install 'PyPyDispatcher>=2.1.0'``. +that the ``PyPyDispatcher`` dependency wasn't installed. To fix this issue, run +``pip install 'PyPyDispatcher>=2.1.0'``. .. _intro-install-troubleshooting: diff --git a/docs/intro/overview.rst b/docs/intro/overview.rst index 488e1c2e3..5937c5860 100644 --- a/docs/intro/overview.rst +++ b/docs/intro/overview.rst @@ -83,7 +83,7 @@ While this enables you to do very fast crawls (sending multiple concurrent requests at the same time, in a fault-tolerant way) Scrapy also gives you control over the politeness of the crawl through :ref:`a few settings `. You can do things like setting a download delay between -each request, limiting the amount of concurrent requests per domain or per IP, and +each request, limiting the amount of concurrent requests per domain, and even :ref:`using an auto-throttling extension ` that tries to figure these settings out automatically. diff --git a/docs/topics/asyncio.rst b/docs/topics/asyncio.rst index 17b322b5a..3d4ebe088 100644 --- a/docs/topics/asyncio.rst +++ b/docs/topics/asyncio.rst @@ -5,7 +5,7 @@ asyncio ======= Scrapy supports :mod:`asyncio` natively. New projects created with -:command:`scrapy startproject` have asyncio enabled by default, and you can use +:command:`startproject` have asyncio enabled by default, and you can use :mod:`asyncio` and :mod:`asyncio`-powered libraries in any :doc:`coroutine `. @@ -18,7 +18,7 @@ no additional setup is needed. Configuring the asyncio reactor =============================== -New projects generated with :command:`scrapy startproject` have the asyncio +New projects generated with :command:`startproject` have the asyncio reactor configured by default. No manual setup is needed. The :setting:`TWISTED_REACTOR` setting controls which Twisted reactor Scrapy diff --git a/docs/topics/autothrottle.rst b/docs/topics/autothrottle.rst index 787ef842a..4f28019da 100644 --- a/docs/topics/autothrottle.rst +++ b/docs/topics/autothrottle.rst @@ -75,7 +75,7 @@ AutoThrottle algorithm adjusts download delays based on the following rules: .. _download-latency: In Scrapy, the download latency is measured as the time elapsed between -establishing the TCP connection and receiving the HTTP headers. +sending the request and receiving the HTTP headers. Note that these latencies are very hard to measure accurately in a cooperative multitasking environment because Scrapy may be busy processing a spider diff --git a/docs/topics/commands.rst b/docs/topics/commands.rst index 270d9859d..ee2c3a3cd 100644 --- a/docs/topics/commands.rst +++ b/docs/topics/commands.rst @@ -309,11 +309,25 @@ Usage examples:: * parse_item $ scrapy check - [FAILED] first_spider:parse_item - >>> 'RetailPricex' field is missing + F.F. + ====================================================================== + FAIL: [first_spider] parse (@returns post-hook) + ---------------------------------------------------------------------- + Traceback (most recent call last): + ... + scrapy.exceptions.ContractFail: Returned 92 requests, expected 0..4 - [FAILED] first_spider:parse - >>> Returned 92 requests, expected 0..4 + ====================================================================== + FAIL: [first_spider] parse_item (@scrapes post-hook) + ---------------------------------------------------------------------- + Traceback (most recent call last): + ... + scrapy.exceptions.ContractFail: Missing fields: RetailPricex + + ---------------------------------------------------------------------- + Ran 4 contracts in 0.174s + + FAILED (failures=2) .. skip: end @@ -377,7 +391,7 @@ Supported options: * ``--spider=SPIDER``: bypass spider autodetection and force use of specific spider -* ``--headers``: print the response's HTTP headers instead of the response's body +* ``--headers``: print the request's and response's HTTP headers instead of the response's body * ``--no-redirect``: do not follow HTTP 3xx redirects (default is to follow them) @@ -387,15 +401,19 @@ Usage examples:: [ ... html content here ... ] $ scrapy fetch --nolog --headers http://www.example.com/ - {'Accept-Ranges': ['bytes'], - 'Age': ['1263 '], - 'Connection': ['close '], - 'Content-Length': ['596'], - 'Content-Type': ['text/html; charset=UTF-8'], - 'Date': ['Wed, 18 Aug 2010 23:59:46 GMT'], - 'Etag': ['"573c1-254-48c9c87349680"'], - 'Last-Modified': ['Fri, 30 Jul 2010 15:30:18 GMT'], - 'Server': ['Apache/2.2.3 (CentOS)']} + > Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 + > Accept-Language: en + > User-Agent: Scrapy/2.16.0 (+https://scrapy.org) + > Accept-Encoding: gzip, deflate, br + > + < Date: Wed, 08 Jul 2026 06:15:01 GMT + < Content-Type: text/html + < Server: cloudflare + < Last-Modified: Wed, 01 Jul 2026 17:50:18 GMT + < Allow: GET, HEAD + < Cf-Cache-Status: HIT + < Age: 8184 + < Cf-Ray: a17cf3b80eddf141-DME .. command:: view @@ -605,7 +623,10 @@ shouldn't matter to the user running the command, but when the user :ref:`needs a non-default Twisted reactor `, it may be important. Scrapy decides which of these two classes to use based on the value of the -:setting:`TWISTED_REACTOR` setting. If the setting value is the default one +:setting:`TWISTED_REACTOR` and :setting:`TWISTED_REACTOR_ENABLED` settings. +With :setting:`TWISTED_REACTOR_ENABLED` set to ``False`` it will use +:class:`~scrapy.crawler.AsyncCrawlerProcess`. Otherwise, if the +:setting:`TWISTED_REACTOR` value is the default one (``'twisted.internet.asyncioreactor.AsyncioSelectorReactor'``), :class:`~scrapy.crawler.AsyncCrawlerProcess` will be used, otherwise :class:`~scrapy.crawler.CrawlerProcess` will be used. The :ref:`spider settings diff --git a/docs/topics/coroutines.rst b/docs/topics/coroutines.rst index 8aaf0aa31..9dcd9d69c 100644 --- a/docs/topics/coroutines.rst +++ b/docs/topics/coroutines.rst @@ -16,7 +16,7 @@ Supported callables The following callables may be defined as coroutines using ``async def``, and hence use coroutine syntax (e.g. ``await``, ``async for``, ``async with``): -- The :meth:`~scrapy.spiders.Spider.start` spider method, which *must* be +- The :meth:`~scrapy.Spider.start` spider method, which *must* be defined as an :term:`asynchronous generator`. .. versionadded:: 2.13 @@ -204,13 +204,15 @@ This means you can use many useful Python libraries providing such code: Common use cases for asynchronous code include: * requesting data from websites, databases and other services (in - :meth:`~scrapy.spiders.Spider.start`, callbacks, pipelines and + :meth:`~scrapy.Spider.start`, callbacks, pipelines and middlewares); * storing data in databases (in pipelines and middlewares); * delaying the spider initialization until some external event (in the :signal:`spider_opened` handler); -* calling asynchronous Scrapy methods like :meth:`ExecutionEngine.download` - (see :ref:`the screenshot pipeline example`). +* calling asynchronous Scrapy methods like + :meth:`ExecutionEngine.download_async() + ` (see :ref:`the + screenshot pipeline example `). .. _aio-libs: https://github.com/aio-libs diff --git a/docs/topics/download-handlers.rst b/docs/topics/download-handlers.rst index 9bbfee9e2..5fbc5df06 100644 --- a/docs/topics/download-handlers.rst +++ b/docs/topics/download-handlers.rst @@ -89,7 +89,7 @@ the following API: If ``True``, the handler will only be instantiated when the first request handled by it needs to be downloaded. - .. method:: download_request(request: Request) -> Response: + .. method:: download_request(request: Request) -> Response :async: Download the given request and return a response. diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst index 383f331b5..096f87edd 100644 --- a/docs/topics/downloader-middleware.rst +++ b/docs/topics/downloader-middleware.rst @@ -591,8 +591,8 @@ In order to use your storage backend, set: HTTPCache middleware settings ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -The :class:`HttpCacheMiddleware` can be configured through the following -settings: +:class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware` can be +configured through the following settings: .. setting:: HTTPCACHE_ENABLED @@ -868,9 +868,9 @@ OffsiteMiddleware .. reqmeta:: allow_offsite If the request has the :attr:`~scrapy.Request.dont_filter` attribute set to - ``True`` or :attr:`Request.meta` has ``allow_offsite`` set to ``True``, then - the OffsiteMiddleware will allow the request even if its domain is not listed - in allowed domains. + ``True`` or :attr:`Request.meta ` has ``allow_offsite`` + set to ``True``, then the OffsiteMiddleware will allow the request even if + its domain is not listed in allowed domains. RedirectMiddleware ------------------ @@ -1015,17 +1015,6 @@ RetryMiddleware A middleware to retry failed requests that are potentially caused by temporary problems such as a connection timeout or HTTP 500 error. -Failed pages are collected on the scraping process and rescheduled at the -end, once the spider has finished crawling all regular (non failed) pages. - -The :class:`RetryMiddleware` can be configured through the following -settings (see the settings documentation for more info): - -* :setting:`RETRY_ENABLED` -* :setting:`RETRY_TIMES` -* :setting:`RETRY_HTTP_CODES` -* :setting:`RETRY_EXCEPTIONS` - .. reqmeta:: dont_retry If :attr:`Request.meta ` has ``dont_retry`` key diff --git a/docs/topics/exceptions.rst b/docs/topics/exceptions.rst index 4c56a0f14..45acb1763 100644 --- a/docs/topics/exceptions.rst +++ b/docs/topics/exceptions.rst @@ -12,7 +12,8 @@ Exceptions Built-in Exceptions reference ============================= -Here's a list of all exceptions included in Scrapy and their usage. +Here's a list of all exceptions included in Scrapy and their usage, except for +the :ref:`download handler exceptions `. CloseSpider @@ -71,7 +72,8 @@ remain disabled. Those components include: - Downloader middlewares - Spider middlewares -The exception must be raised in the component's ``__init__`` method. +The exception must be raised in the component's ``__init__()`` or +``from_crawler()`` method. NotSupported ------------ diff --git a/docs/topics/feed-exports.rst b/docs/topics/feed-exports.rst index ef1abe747..1bab3bfc2 100644 --- a/docs/topics/feed-exports.rst +++ b/docs/topics/feed-exports.rst @@ -166,7 +166,7 @@ The feeds are stored in the local filesystem. - Required external libraries: none Note that for the local filesystem storage (only) you can omit the scheme if -you specify an absolute path like ``/tmp/export.csv`` (Unix systems only). +you specify a path (e.g. ``/tmp/export.csv``). Alternatively you can also use a :class:`pathlib.Path` object. .. _topics-feed-storage-ftp: @@ -762,8 +762,8 @@ The function signature should be as follows: :param spider: source spider of the feed items :type spider: scrapy.Spider - .. caution:: The function should return a new dictionary, modifying - the received ``params`` in-place is deprecated. + .. caution:: The function must return a new dictionary instead of modifying + the received ``params`` in-place. For example, to include the :attr:`name ` of the source spider in the feed URI: diff --git a/docs/topics/item-pipeline.rst b/docs/topics/item-pipeline.rst index f1fa463d8..951c0f485 100644 --- a/docs/topics/item-pipeline.rst +++ b/docs/topics/item-pipeline.rst @@ -121,7 +121,7 @@ Write items to MongoDB In this example we'll write items to MongoDB_ using pymongo_. MongoDB address and database name are specified in Scrapy settings; -MongoDB collection is named after item class. +MongoDB collection is specified in a class attribute. The main point of this example is to show how to :ref:`get the crawler ` and how to clean up the resources properly. diff --git a/docs/topics/items.rst b/docs/topics/items.rst index b21d6b903..0892dd839 100644 --- a/docs/topics/items.rst +++ b/docs/topics/items.rst @@ -23,7 +23,8 @@ Item Types Scrapy supports the following types of items, via the `itemadapter`_ library: :ref:`dictionaries `, :ref:`Item objects `, -:ref:`dataclass objects `, and :ref:`attrs objects `. +:ref:`dataclass objects `, :ref:`attrs objects ` +and :ref:`Pydantic models `. .. _itemadapter: https://github.com/scrapy/itemadapter @@ -61,8 +62,8 @@ its ``__init__`` method. :class:`Item` also allows the defining of field metadata, which can be used to :ref:`customize serialization `. -:mod:`trackref` tracks :class:`Item` objects to help find memory leaks -(see :ref:`topics-leaks-trackrefs`). +:mod:`scrapy.utils.trackref` tracks :class:`Item` objects to help find memory +leaks (see :ref:`topics-leaks-trackrefs`). Example: @@ -262,7 +263,7 @@ Creating items >>> product = Product(name="Desktop PC", price=1000) >>> print(product) - Product(name='Desktop PC', price=1000) + {'name': 'Desktop PC', 'price': 1000} Getting field values @@ -381,7 +382,7 @@ Creating items from dicts: .. code-block:: pycon >>> Product({"name": "Laptop PC", "price": 1500}) - Product(price=1500, name='Laptop PC') + {'name': 'Laptop PC', 'price': 1500} >>> Product({"name": "Laptop PC", "lala": 1500}) # warning: unknown field in dict Traceback (most recent call last): diff --git a/docs/topics/jobs.rst b/docs/topics/jobs.rst index 8d6b2458b..c9916110d 100644 --- a/docs/topics/jobs.rst +++ b/docs/topics/jobs.rst @@ -151,8 +151,8 @@ Where: - :class:`~scrapy.pqueues.ScrapyPriorityQueue` creates the ``{priority}{s?}`` directories. -- :class:`scrapy.squeues.PickleLifoDiskQueue`, a subclass of - :class:`queuelib.LifoDiskQueue` that uses :mod:`pickle` to serialize +- :class:`scrapy.squeues.PickleFifoDiskQueue`, a subclass of + :class:`queuelib.FifoDiskQueue` that uses :mod:`pickle` to serialize :class:`dict` representations of :class:`scrapy.Request` objects, creates the ``info.json`` and ``q{00000}`` files. diff --git a/docs/topics/leaks.rst b/docs/topics/leaks.rst index 70d563fdd..3337cc5d0 100644 --- a/docs/topics/leaks.rst +++ b/docs/topics/leaks.rst @@ -62,9 +62,9 @@ Debugging memory leaks with ``trackref`` .. skip: start -:mod:`trackref` is a module provided by Scrapy to debug the most common cases of -memory leaks. It basically tracks the references to all live Request, -Response, Item, Spider and Selector objects. +:mod:`scrapy.utils.trackref` is a module provided by Scrapy to debug the most +common cases of memory leaks. It basically tracks the references to all live +Request, Response, Item, Spider and Selector objects. You can enter the telnet console and inspect how many objects (of the classes mentioned above) are currently alive using the ``prefs()`` function which is an @@ -164,7 +164,7 @@ Too many spiders? ----------------- If your project has too many spiders executed in parallel, -the output of :func:`prefs` can be difficult to read. +the output of ``prefs()`` can be difficult to read. For this reason, that function has a ``ignore`` argument which can be used to ignore a particular class (and all its subclasses). For example, this won't show any live references to spiders: diff --git a/docs/topics/loaders.rst b/docs/topics/loaders.rst index 801dd6583..765e7f6d7 100644 --- a/docs/topics/loaders.rst +++ b/docs/topics/loaders.rst @@ -264,7 +264,7 @@ metadata. Here is an example: >>> il.add_value("name", ["Welcome to my", "website"]) >>> il.add_value("price", ["€", "1000"]) >>> il.load_item() - {'name': 'Welcome to my website', 'price': '1000'} + Product(name='Welcome to my website', price='1000') .. skip: end diff --git a/docs/topics/media-pipeline.rst b/docs/topics/media-pipeline.rst index 04a744b0e..b7a4f561e 100644 --- a/docs/topics/media-pipeline.rst +++ b/docs/topics/media-pipeline.rst @@ -41,11 +41,10 @@ this: 2. The item is returned from the spider and goes to the item pipeline. 3. When the item reaches the :class:`FilesPipeline`, the URLs in the - ``file_urls`` field are scheduled for download using the standard - Scrapy scheduler and downloader (which means the scheduler and downloader - middlewares are reused), but with a higher priority, processing them before other - pages are scraped. The item remains "locked" at that particular pipeline stage - until the files have finish downloading (or fail for some reason). + ``file_urls`` field are downloaded using the standard Scrapy downloader + (which means the downloader middlewares are used, but the spider middlewares + aren't). The item remains "locked" at that particular pipeline stage until + the files have finished downloading (or failed for some reason). 4. When the files are downloaded, another field (``files``) will be populated with the results. This field will contain a list of dicts with information @@ -548,10 +547,9 @@ See here the methods that you can override in your custom Files Pipeline: .. method:: FilesPipeline.get_media_requests(item, info) - As seen on the workflow, the pipeline will get the URLs of the images to - download from the item. In order to do this, you can override the - :meth:`~get_media_requests` method and return a Request for each - file URL: + As seen on the workflow, the pipeline will get the requests for the files + to download from the item by calling this method. You can override it to + change what requests are returned: .. code-block:: python @@ -591,8 +589,9 @@ See here the methods that you can override in your custom Files Pipeline: * ``downloaded`` - file was downloaded. * ``uptodate`` - file was not downloaded, as it was downloaded recently, according to the file expiration policy. - * ``cached`` - file was already scheduled for download, by another item - sharing the same file. + * ``cached`` - file was taken from a cache (the response has a + ``"cached"`` flag, e.g. from + :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware`). The list of tuples received by :meth:`~item_completed` is guaranteed to retain the same order of the requests returned from the @@ -619,9 +618,6 @@ See here the methods that you can override in your custom Files Pipeline: (False, Failure(...)), ] - By default the :meth:`get_media_requests` method returns ``None`` which - means there are no files to download for the item. - .. method:: FilesPipeline.item_completed(results, item, info) The :meth:`FilesPipeline.item_completed` method called when all file diff --git a/docs/topics/practices.rst b/docs/topics/practices.rst index e7faf48c4..216b7e64f 100644 --- a/docs/topics/practices.rst +++ b/docs/topics/practices.rst @@ -17,8 +17,10 @@ Run Scrapy from a script You can use the :ref:`API ` to run Scrapy from a script, instead of the typical way of running Scrapy via ``scrapy crawl``. -Remember that Scrapy is built on top of the Twisted -asynchronous networking library, so you need to run it inside the Twisted reactor. +Remember that Scrapy requires a Twisted reactor or (with +:setting:`TWISTED_REACTOR_ENABLED` set to ``False``) an asyncio event loop, so +you need to run one of those in your script for it to work (helpers described +below can do it for you). The first utility you can use to run your spiders is :class:`scrapy.crawler.AsyncCrawlerProcess` or diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rst index 7796755af..a5f799444 100644 --- a/docs/topics/request-response.rst +++ b/docs/topics/request-response.rst @@ -1101,8 +1101,8 @@ Response objects The IP address of the server from which the Response originated. - This attribute is currently only populated by the HTTP 1.1 download - handler, i.e. for ``http(s)`` responses. For other handlers, + This attribute is currently only populated by the HTTP download + handlers, i.e. for ``http(s)`` responses. For other handlers, :attr:`ip_address` is always ``None``. .. attribute:: Response.protocol diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst index 6084fdb5b..a592f8bc5 100644 --- a/docs/topics/settings.rst +++ b/docs/topics/settings.rst @@ -418,7 +418,9 @@ Default: ``None`` Import path of a given ``asyncio`` event loop class. -If the asyncio reactor is enabled (see :setting:`TWISTED_REACTOR`) this setting can be used to specify the +If the asyncio reactor is enabled (see :setting:`TWISTED_REACTOR`) or when +:ref:`running Scrapy without a reactor ` this setting +can be used to specify the asyncio event loop to be used with it. Set the setting to the import path of the desired asyncio event loop class. If the setting is set to ``None`` the default asyncio event loop will be used. @@ -554,6 +556,8 @@ performed to any single domain. See also: :ref:`topics-autothrottle` and its :setting:`AUTOTHROTTLE_TARGET_CONCURRENCY` option. +It is possible to change this setting per domain by using +:setting:`DOWNLOAD_SLOTS`. .. setting:: DEFAULT_DROPITEM_LOG_LEVEL @@ -917,9 +921,8 @@ desired. This delay can be set per spider using :attr:`download_delay` spider attribute. -It is also possible to change this setting per domain, although it requires -non-trivial code. See the implementation of the :ref:`AutoThrottle -` extension for an example. +It is possible to change this setting per domain by using +:setting:`DOWNLOAD_SLOTS`. .. setting:: DOWNLOAD_BIND_ADDRESS @@ -1358,14 +1361,19 @@ FORCE_CRAWLER_PROCESS Default: ``False`` If ``False``, :ref:`Scrapy commands that need a CrawlerProcess -` will decide between using +`, when :setting:`TWISTED_REACTOR_ENABLED` +is set to ``True``, will decide between using :class:`scrapy.crawler.AsyncCrawlerProcess` and :class:`scrapy.crawler.CrawlerProcess` based on the value of the :setting:`TWISTED_REACTOR` setting, but ignoring its value in :ref:`per-spider settings `. If ``True``, these commands will always use -:class:`~scrapy.crawler.CrawlerProcess`. +:class:`~scrapy.crawler.CrawlerProcess` when :setting:`TWISTED_REACTOR_ENABLED` +is set to ``True``. + +When :setting:`TWISTED_REACTOR_ENABLED` is set to ``False``, +:class:`~scrapy.crawler.AsyncCrawlerProcess` will be used in all cases. Set this to ``True`` if you want to set :setting:`TWISTED_REACTOR` to a non-default value in :ref:`per-spider settings `. @@ -1746,6 +1754,9 @@ The randomization policy is the same used by `wget`_ ``--random-wait`` option. If :setting:`DOWNLOAD_DELAY` is zero this option has no effect. +It is possible to change this setting per domain by using +:setting:`DOWNLOAD_SLOTS`. + .. _wget: https://www.gnu.org/software/wget/manual/wget.html .. setting:: REACTOR_THREADPOOL_MAXSIZE @@ -2159,6 +2170,9 @@ Default: ``"twisted.internet.asyncioreactor.AsyncioSelectorReactor"`` Import path of a given :mod:`~twisted.internet.reactor`. +.. note:: + This setting has no effect when :setting:`TWISTED_REACTOR_ENABLED` is ``False``. + Scrapy will install this reactor if no other reactor is installed yet, such as when the ``scrapy`` CLI program is invoked or when using the :class:`~scrapy.crawler.AsyncCrawlerProcess` class or the diff --git a/docs/topics/signals.rst b/docs/topics/signals.rst index bd14f5993..531232149 100644 --- a/docs/topics/signals.rst +++ b/docs/topics/signals.rst @@ -454,7 +454,7 @@ bytes_received .. signal:: bytes_received .. function:: bytes_received(data, request, spider) - Sent by the HTTP 1.1 and S3 download handlers when a group of bytes is + Sent by some download handlers when a group of bytes is received for a specific request. This signal might be fired multiple times for the same request, with partial data each time. For instance, a possible scenario for a 25 kb response would be two signals fired @@ -482,7 +482,7 @@ headers_received .. signal:: headers_received .. function:: headers_received(headers, body_length, request, spider) - Sent by the HTTP 1.1 and S3 download handlers when the response headers are + Sent by some download handlers when the response headers are available for a given request, before downloading any additional content. Handlers for this signal can stop the download of a response while it diff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rst index bb5952826..887da0a15 100644 --- a/docs/topics/spiders.rst +++ b/docs/topics/spiders.rst @@ -614,7 +614,7 @@ XMLFeedSpider A string which defines the iterator to use. It can be either: - - ``'iternodes'`` - a fast iterator based on regular expressions + - ``'iternodes'`` - a fast iterator based on ``lxml`` - ``'html'`` - an iterator which uses :class:`~scrapy.Selector`. Keep in mind this uses DOM parsing and must load all DOM in memory diff --git a/docs/topics/stats.rst b/docs/topics/stats.rst index ac714eacb..028fb009d 100644 --- a/docs/topics/stats.rst +++ b/docs/topics/stats.rst @@ -10,8 +10,8 @@ Collector, and can be accessed through the :attr:`~scrapy.crawler.Crawler.stats` attribute of the :ref:`topics-api-crawler`, as illustrated by the examples in the :ref:`topics-stats-usecases` section below. -However, the Stats Collector is always available, so you can always import it -in your module and use its API (to increment or set new stat keys), regardless +The Stats Collector API is always available, so you can always use it (to +increment or set new stat keys), regardless of whether the stats collection is enabled or not. If it's disabled, the API will still work but it won't collect anything. This is aimed at simplifying the stats collector usage: you should spend no more than one line of code for @@ -21,9 +21,6 @@ using the Stats Collector from. Another feature of the Stats Collector is that it's very efficient (when enabled) and extremely efficient (almost unnoticeable) when disabled. -The Stats Collector keeps a stats table per open spider which is automatically -opened when the spider is opened, and closed when the spider is closed. - .. _topics-stats-usecases: Common Stats Collector uses @@ -87,13 +84,13 @@ Get all stats: Available Stats Collectors ========================== +.. currentmodule:: scrapy.statscollectors + Besides the basic :class:`StatsCollector` there are other Stats Collectors available in Scrapy which extend the basic Stats Collector. You can select which Stats Collector to use through the :setting:`STATS_CLASS` setting. The default Stats Collector used is the :class:`MemoryStatsCollector`. -.. currentmodule:: scrapy.statscollectors - MemoryStatsCollector -------------------- @@ -102,7 +99,7 @@ MemoryStatsCollector A simple stats collector that keeps the stats of the last scraping run (for each spider) in memory, after they're closed. The stats can be accessed through the :attr:`spider_stats` attribute, which is a dict keyed by spider - domain name. + name. This is the default Stats Collector used in Scrapy. diff --git a/docs/topics/telnetconsole.rst b/docs/topics/telnetconsole.rst index 72188c03f..54379b4b9 100644 --- a/docs/topics/telnetconsole.rst +++ b/docs/topics/telnetconsole.rst @@ -107,8 +107,8 @@ Here are some example tasks you can do with the telnet console: View engine status ------------------ -You can use the ``est()`` method of the Scrapy engine to quickly show its state -using the telnet console:: +You can use the ``est()`` method provided by the console to quickly show the +engine status:: telnet localhost 6023 >>> est() diff --git a/scrapy/core/scheduler.py b/scrapy/core/scheduler.py index 82e31b90b..42c517222 100644 --- a/scrapy/core/scheduler.py +++ b/scrapy/core/scheduler.py @@ -289,11 +289,11 @@ class Scheduler(BaseScheduler): :param dqclass: A class to be used as persistent request queue. The value for the :setting:`SCHEDULER_DISK_QUEUE` setting is used by default. - :type dqclass: class + :type dqclass: type :param mqclass: A class to be used as non-persistent request queue. The value for the :setting:`SCHEDULER_MEMORY_QUEUE` setting is used by default. - :type mqclass: class + :type mqclass: type :param logunser: A boolean that indicates whether or not unserializable requests should be logged. The value for the :setting:`SCHEDULER_DEBUG` setting is used by default. @@ -306,7 +306,7 @@ class Scheduler(BaseScheduler): :param pqclass: A class to be used as priority queue for requests. The value for the :setting:`SCHEDULER_PRIORITY_QUEUE` setting is used by default. - :type pqclass: class + :type pqclass: type :param crawler: The crawler object corresponding to the current crawl. :type crawler: :class:`scrapy.crawler.Crawler` diff --git a/scrapy/settings/__init__.py b/scrapy/settings/__init__.py index 932463ba9..27b0f1187 100644 --- a/scrapy/settings/__init__.py +++ b/scrapy/settings/__init__.py @@ -288,10 +288,10 @@ class BaseSettings(MutableMapping[str, Any]): - ``['one', 'two']`` if set to ``'["one", "two"]'`` or ``'one,two'`` :param name: the setting name - :type name: string + :type name: str :param default: the value to return if no setting is found - :type default: any + :type default: object """ value = self.get(name, default) if value is None: