mirror of https://github.com/scrapy/scrapy.git
Assorted docs fixes, part 2. (#7725)
* Assorted docs fixes, part 2. * Second pass. * Address feedback.
This commit is contained in:
Andrey Rakhmatullin
GitHub
parent
b3670369b8
commit
d8ba1571e7
|
|
@ -384,8 +384,9 @@ How to deal with ``<class 'ValueError'>: filedescriptor out of range in select()
|
|||
----------------------------------------------------------------------------------------------
|
||||
|
||||
This issue `has been reported`_ to appear when running broad crawls in macOS, where the default
|
||||
Twisted reactor is :class:`twisted.internet.selectreactor.SelectReactor`. Switching to a
|
||||
different reactor is possible by using the :setting:`TWISTED_REACTOR` setting.
|
||||
Twisted reactor was :class:`twisted.internet.selectreactor.SelectReactor` at that time.
|
||||
If you have switched to this reactor using the :setting:`TWISTED_REACTOR` setting you can switch
|
||||
to a different one in the same way.
|
||||
|
||||
|
||||
.. _faq-stop-response-download:
|
||||
|
|
|
|||
|
|
@ -230,8 +230,8 @@ Installing Scrapy with PyPy on Windows is not tested.
|
|||
You can check that Scrapy is installed correctly by running ``scrapy bench``.
|
||||
If this command gives errors such as
|
||||
``TypeError: ... got 2 unexpected keyword arguments``, this means
|
||||
that setuptools was unable to pick up one PyPy-specific dependency.
|
||||
To fix this issue, run ``pip install 'PyPyDispatcher>=2.1.0'``.
|
||||
that the ``PyPyDispatcher`` dependency wasn't installed. To fix this issue, run
|
||||
``pip install 'PyPyDispatcher>=2.1.0'``.
|
||||
|
||||
|
||||
.. _intro-install-troubleshooting:
|
||||
|
|
|
|||
|
|
@ -83,7 +83,7 @@ While this enables you to do very fast crawls (sending multiple concurrent
|
|||
requests at the same time, in a fault-tolerant way) Scrapy also gives you
|
||||
control over the politeness of the crawl through :ref:`a few settings
|
||||
<topics-settings-ref>`. You can do things like setting a download delay between
|
||||
each request, limiting the amount of concurrent requests per domain or per IP, and
|
||||
each request, limiting the amount of concurrent requests per domain, and
|
||||
even :ref:`using an auto-throttling extension <topics-autothrottle>` that tries
|
||||
to figure these settings out automatically.
|
||||
|
||||
|
|
|
|||
|
|
@ -5,7 +5,7 @@ asyncio
|
|||
=======
|
||||
|
||||
Scrapy supports :mod:`asyncio` natively. New projects created with
|
||||
:command:`scrapy startproject` have asyncio enabled by default, and you can use
|
||||
:command:`startproject` have asyncio enabled by default, and you can use
|
||||
:mod:`asyncio` and :mod:`asyncio`-powered libraries in any :doc:`coroutine
|
||||
<coroutines>`.
|
||||
|
||||
|
|
@ -18,7 +18,7 @@ no additional setup is needed.
|
|||
Configuring the asyncio reactor
|
||||
===============================
|
||||
|
||||
New projects generated with :command:`scrapy startproject` have the asyncio
|
||||
New projects generated with :command:`startproject` have the asyncio
|
||||
reactor configured by default. No manual setup is needed.
|
||||
|
||||
The :setting:`TWISTED_REACTOR` setting controls which Twisted reactor Scrapy
|
||||
|
|
|
|||
|
|
@ -75,7 +75,7 @@ AutoThrottle algorithm adjusts download delays based on the following rules:
|
|||
.. _download-latency:
|
||||
|
||||
In Scrapy, the download latency is measured as the time elapsed between
|
||||
establishing the TCP connection and receiving the HTTP headers.
|
||||
sending the request and receiving the HTTP headers.
|
||||
|
||||
Note that these latencies are very hard to measure accurately in a cooperative
|
||||
multitasking environment because Scrapy may be busy processing a spider
|
||||
|
|
|
|||
|
|
@ -309,11 +309,25 @@ Usage examples::
|
|||
* parse_item
|
||||
|
||||
$ scrapy check
|
||||
[FAILED] first_spider:parse_item
|
||||
>>> 'RetailPricex' field is missing
|
||||
F.F.
|
||||
======================================================================
|
||||
FAIL: [first_spider] parse (@returns post-hook)
|
||||
----------------------------------------------------------------------
|
||||
Traceback (most recent call last):
|
||||
...
|
||||
scrapy.exceptions.ContractFail: Returned 92 requests, expected 0..4
|
||||
|
||||
[FAILED] first_spider:parse
|
||||
>>> Returned 92 requests, expected 0..4
|
||||
======================================================================
|
||||
FAIL: [first_spider] parse_item (@scrapes post-hook)
|
||||
----------------------------------------------------------------------
|
||||
Traceback (most recent call last):
|
||||
...
|
||||
scrapy.exceptions.ContractFail: Missing fields: RetailPricex
|
||||
|
||||
----------------------------------------------------------------------
|
||||
Ran 4 contracts in 0.174s
|
||||
|
||||
FAILED (failures=2)
|
||||
|
||||
.. skip: end
|
||||
|
||||
|
|
@ -377,7 +391,7 @@ Supported options:
|
|||
|
||||
* ``--spider=SPIDER``: bypass spider autodetection and force use of specific spider
|
||||
|
||||
* ``--headers``: print the response's HTTP headers instead of the response's body
|
||||
* ``--headers``: print the request's and response's HTTP headers instead of the response's body
|
||||
|
||||
* ``--no-redirect``: do not follow HTTP 3xx redirects (default is to follow them)
|
||||
|
||||
|
|
@ -387,15 +401,19 @@ Usage examples::
|
|||
[ ... html content here ... ]
|
||||
|
||||
$ scrapy fetch --nolog --headers http://www.example.com/
|
||||
{'Accept-Ranges': ['bytes'],
|
||||
'Age': ['1263 '],
|
||||
'Connection': ['close '],
|
||||
'Content-Length': ['596'],
|
||||
'Content-Type': ['text/html; charset=UTF-8'],
|
||||
'Date': ['Wed, 18 Aug 2010 23:59:46 GMT'],
|
||||
'Etag': ['"573c1-254-48c9c87349680"'],
|
||||
'Last-Modified': ['Fri, 30 Jul 2010 15:30:18 GMT'],
|
||||
'Server': ['Apache/2.2.3 (CentOS)']}
|
||||
> Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
|
||||
> Accept-Language: en
|
||||
> User-Agent: Scrapy/2.16.0 (+https://scrapy.org)
|
||||
> Accept-Encoding: gzip, deflate, br
|
||||
>
|
||||
< Date: Wed, 08 Jul 2026 06:15:01 GMT
|
||||
< Content-Type: text/html
|
||||
< Server: cloudflare
|
||||
< Last-Modified: Wed, 01 Jul 2026 17:50:18 GMT
|
||||
< Allow: GET, HEAD
|
||||
< Cf-Cache-Status: HIT
|
||||
< Age: 8184
|
||||
< Cf-Ray: a17cf3b80eddf141-DME
|
||||
|
||||
.. command:: view
|
||||
|
||||
|
|
@ -605,7 +623,10 @@ shouldn't matter to the user running the command, but when the user :ref:`needs
|
|||
a non-default Twisted reactor <disable-asyncio>`, it may be important.
|
||||
|
||||
Scrapy decides which of these two classes to use based on the value of the
|
||||
:setting:`TWISTED_REACTOR` setting. If the setting value is the default one
|
||||
:setting:`TWISTED_REACTOR` and :setting:`TWISTED_REACTOR_ENABLED` settings.
|
||||
With :setting:`TWISTED_REACTOR_ENABLED` set to ``False`` it will use
|
||||
:class:`~scrapy.crawler.AsyncCrawlerProcess`. Otherwise, if the
|
||||
:setting:`TWISTED_REACTOR` value is the default one
|
||||
(``'twisted.internet.asyncioreactor.AsyncioSelectorReactor'``),
|
||||
:class:`~scrapy.crawler.AsyncCrawlerProcess` will be used, otherwise
|
||||
:class:`~scrapy.crawler.CrawlerProcess` will be used. The :ref:`spider settings
|
||||
|
|
|
|||
|
|
@ -16,7 +16,7 @@ Supported callables
|
|||
The following callables may be defined as coroutines using ``async def``, and
|
||||
hence use coroutine syntax (e.g. ``await``, ``async for``, ``async with``):
|
||||
|
||||
- The :meth:`~scrapy.spiders.Spider.start` spider method, which *must* be
|
||||
- The :meth:`~scrapy.Spider.start` spider method, which *must* be
|
||||
defined as an :term:`asynchronous generator`.
|
||||
|
||||
.. versionadded:: 2.13
|
||||
|
|
@ -204,13 +204,15 @@ This means you can use many useful Python libraries providing such code:
|
|||
Common use cases for asynchronous code include:
|
||||
|
||||
* requesting data from websites, databases and other services (in
|
||||
:meth:`~scrapy.spiders.Spider.start`, callbacks, pipelines and
|
||||
:meth:`~scrapy.Spider.start`, callbacks, pipelines and
|
||||
middlewares);
|
||||
* storing data in databases (in pipelines and middlewares);
|
||||
* delaying the spider initialization until some external event (in the
|
||||
:signal:`spider_opened` handler);
|
||||
* calling asynchronous Scrapy methods like :meth:`ExecutionEngine.download`
|
||||
(see :ref:`the screenshot pipeline example<ScreenshotPipeline>`).
|
||||
* calling asynchronous Scrapy methods like
|
||||
:meth:`ExecutionEngine.download_async()
|
||||
<scrapy.core.engine.ExecutionEngine.download_async>` (see :ref:`the
|
||||
screenshot pipeline example <ScreenshotPipeline>`).
|
||||
|
||||
.. _aio-libs: https://github.com/aio-libs
|
||||
|
||||
|
|
|
|||
|
|
@ -89,7 +89,7 @@ the following API:
|
|||
If ``True``, the handler will only be instantiated when the first
|
||||
request handled by it needs to be downloaded.
|
||||
|
||||
.. method:: download_request(request: Request) -> Response:
|
||||
.. method:: download_request(request: Request) -> Response
|
||||
:async:
|
||||
|
||||
Download the given request and return a response.
|
||||
|
|
|
|||
|
|
@ -591,8 +591,8 @@ In order to use your storage backend, set:
|
|||
HTTPCache middleware settings
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The :class:`HttpCacheMiddleware` can be configured through the following
|
||||
settings:
|
||||
:class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware` can be
|
||||
configured through the following settings:
|
||||
|
||||
.. setting:: HTTPCACHE_ENABLED
|
||||
|
||||
|
|
@ -868,9 +868,9 @@ OffsiteMiddleware
|
|||
.. reqmeta:: allow_offsite
|
||||
|
||||
If the request has the :attr:`~scrapy.Request.dont_filter` attribute set to
|
||||
``True`` or :attr:`Request.meta` has ``allow_offsite`` set to ``True``, then
|
||||
the OffsiteMiddleware will allow the request even if its domain is not listed
|
||||
in allowed domains.
|
||||
``True`` or :attr:`Request.meta <scrapy.Request.meta>` has ``allow_offsite``
|
||||
set to ``True``, then the OffsiteMiddleware will allow the request even if
|
||||
its domain is not listed in allowed domains.
|
||||
|
||||
RedirectMiddleware
|
||||
------------------
|
||||
|
|
@ -1015,17 +1015,6 @@ RetryMiddleware
|
|||
A middleware to retry failed requests that are potentially caused by
|
||||
temporary problems such as a connection timeout or HTTP 500 error.
|
||||
|
||||
Failed pages are collected on the scraping process and rescheduled at the
|
||||
end, once the spider has finished crawling all regular (non failed) pages.
|
||||
|
||||
The :class:`RetryMiddleware` can be configured through the following
|
||||
settings (see the settings documentation for more info):
|
||||
|
||||
* :setting:`RETRY_ENABLED`
|
||||
* :setting:`RETRY_TIMES`
|
||||
* :setting:`RETRY_HTTP_CODES`
|
||||
* :setting:`RETRY_EXCEPTIONS`
|
||||
|
||||
.. reqmeta:: dont_retry
|
||||
|
||||
If :attr:`Request.meta <scrapy.Request.meta>` has ``dont_retry`` key
|
||||
|
|
|
|||
|
|
@ -12,7 +12,8 @@ Exceptions
|
|||
Built-in Exceptions reference
|
||||
=============================
|
||||
|
||||
Here's a list of all exceptions included in Scrapy and their usage.
|
||||
Here's a list of all exceptions included in Scrapy and their usage, except for
|
||||
the :ref:`download handler exceptions <download-handlers-exceptions>`.
|
||||
|
||||
|
||||
CloseSpider
|
||||
|
|
@ -71,7 +72,8 @@ remain disabled. Those components include:
|
|||
- Downloader middlewares
|
||||
- Spider middlewares
|
||||
|
||||
The exception must be raised in the component's ``__init__`` method.
|
||||
The exception must be raised in the component's ``__init__()`` or
|
||||
``from_crawler()`` method.
|
||||
|
||||
NotSupported
|
||||
------------
|
||||
|
|
|
|||
|
|
@ -166,7 +166,7 @@ The feeds are stored in the local filesystem.
|
|||
- Required external libraries: none
|
||||
|
||||
Note that for the local filesystem storage (only) you can omit the scheme if
|
||||
you specify an absolute path like ``/tmp/export.csv`` (Unix systems only).
|
||||
you specify a path (e.g. ``/tmp/export.csv``).
|
||||
Alternatively you can also use a :class:`pathlib.Path` object.
|
||||
|
||||
.. _topics-feed-storage-ftp:
|
||||
|
|
@ -762,8 +762,8 @@ The function signature should be as follows:
|
|||
:param spider: source spider of the feed items
|
||||
:type spider: scrapy.Spider
|
||||
|
||||
.. caution:: The function should return a new dictionary, modifying
|
||||
the received ``params`` in-place is deprecated.
|
||||
.. caution:: The function must return a new dictionary instead of modifying
|
||||
the received ``params`` in-place.
|
||||
|
||||
For example, to include the :attr:`name <scrapy.Spider.name>` of the
|
||||
source spider in the feed URI:
|
||||
|
|
|
|||
|
|
@ -121,7 +121,7 @@ Write items to MongoDB
|
|||
|
||||
In this example we'll write items to MongoDB_ using pymongo_.
|
||||
MongoDB address and database name are specified in Scrapy settings;
|
||||
MongoDB collection is named after item class.
|
||||
MongoDB collection is specified in a class attribute.
|
||||
|
||||
The main point of this example is to show how to :ref:`get the crawler
|
||||
<from-crawler>` and how to clean up the resources properly.
|
||||
|
|
|
|||
|
|
@ -23,7 +23,8 @@ Item Types
|
|||
|
||||
Scrapy supports the following types of items, via the `itemadapter`_ library:
|
||||
:ref:`dictionaries <dict-items>`, :ref:`Item objects <item-objects>`,
|
||||
:ref:`dataclass objects <dataclass-items>`, and :ref:`attrs objects <attrs-items>`.
|
||||
:ref:`dataclass objects <dataclass-items>`, :ref:`attrs objects <attrs-items>`
|
||||
and :ref:`Pydantic models <pydantic-items>`.
|
||||
|
||||
.. _itemadapter: https://github.com/scrapy/itemadapter
|
||||
|
||||
|
|
@ -61,8 +62,8 @@ its ``__init__`` method.
|
|||
:class:`Item` also allows the defining of field metadata, which can be used to
|
||||
:ref:`customize serialization <topics-exporters-field-serialization>`.
|
||||
|
||||
:mod:`trackref` tracks :class:`Item` objects to help find memory leaks
|
||||
(see :ref:`topics-leaks-trackrefs`).
|
||||
:mod:`scrapy.utils.trackref` tracks :class:`Item` objects to help find memory
|
||||
leaks (see :ref:`topics-leaks-trackrefs`).
|
||||
|
||||
Example:
|
||||
|
||||
|
|
@ -262,7 +263,7 @@ Creating items
|
|||
|
||||
>>> product = Product(name="Desktop PC", price=1000)
|
||||
>>> print(product)
|
||||
Product(name='Desktop PC', price=1000)
|
||||
{'name': 'Desktop PC', 'price': 1000}
|
||||
|
||||
|
||||
Getting field values
|
||||
|
|
@ -381,7 +382,7 @@ Creating items from dicts:
|
|||
.. code-block:: pycon
|
||||
|
||||
>>> Product({"name": "Laptop PC", "price": 1500})
|
||||
Product(price=1500, name='Laptop PC')
|
||||
{'name': 'Laptop PC', 'price': 1500}
|
||||
|
||||
>>> Product({"name": "Laptop PC", "lala": 1500}) # warning: unknown field in dict
|
||||
Traceback (most recent call last):
|
||||
|
|
|
|||
|
|
@ -151,8 +151,8 @@ Where:
|
|||
- :class:`~scrapy.pqueues.ScrapyPriorityQueue` creates the ``{priority}{s?}``
|
||||
directories.
|
||||
|
||||
- :class:`scrapy.squeues.PickleLifoDiskQueue`, a subclass of
|
||||
:class:`queuelib.LifoDiskQueue` that uses :mod:`pickle` to serialize
|
||||
- :class:`scrapy.squeues.PickleFifoDiskQueue`, a subclass of
|
||||
:class:`queuelib.FifoDiskQueue` that uses :mod:`pickle` to serialize
|
||||
:class:`dict` representations of :class:`scrapy.Request` objects, creates
|
||||
the ``info.json`` and ``q{00000}`` files.
|
||||
|
||||
|
|
|
|||
|
|
@ -62,9 +62,9 @@ Debugging memory leaks with ``trackref``
|
|||
|
||||
.. skip: start
|
||||
|
||||
:mod:`trackref` is a module provided by Scrapy to debug the most common cases of
|
||||
memory leaks. It basically tracks the references to all live Request,
|
||||
Response, Item, Spider and Selector objects.
|
||||
:mod:`scrapy.utils.trackref` is a module provided by Scrapy to debug the most
|
||||
common cases of memory leaks. It basically tracks the references to all live
|
||||
Request, Response, Item, Spider and Selector objects.
|
||||
|
||||
You can enter the telnet console and inspect how many objects (of the classes
|
||||
mentioned above) are currently alive using the ``prefs()`` function which is an
|
||||
|
|
@ -164,7 +164,7 @@ Too many spiders?
|
|||
-----------------
|
||||
|
||||
If your project has too many spiders executed in parallel,
|
||||
the output of :func:`prefs` can be difficult to read.
|
||||
the output of ``prefs()`` can be difficult to read.
|
||||
For this reason, that function has a ``ignore`` argument which can be used to
|
||||
ignore a particular class (and all its subclasses). For
|
||||
example, this won't show any live references to spiders:
|
||||
|
|
|
|||
|
|
@ -264,7 +264,7 @@ metadata. Here is an example:
|
|||
>>> il.add_value("name", ["Welcome to my", "<strong>website</strong>"])
|
||||
>>> il.add_value("price", ["€", "<span>1000</span>"])
|
||||
>>> il.load_item()
|
||||
{'name': 'Welcome to my website', 'price': '1000'}
|
||||
Product(name='Welcome to my website', price='1000')
|
||||
|
||||
.. skip: end
|
||||
|
||||
|
|
|
|||
|
|
@ -41,11 +41,10 @@ this:
|
|||
2. The item is returned from the spider and goes to the item pipeline.
|
||||
|
||||
3. When the item reaches the :class:`FilesPipeline`, the URLs in the
|
||||
``file_urls`` field are scheduled for download using the standard
|
||||
Scrapy scheduler and downloader (which means the scheduler and downloader
|
||||
middlewares are reused), but with a higher priority, processing them before other
|
||||
pages are scraped. The item remains "locked" at that particular pipeline stage
|
||||
until the files have finish downloading (or fail for some reason).
|
||||
``file_urls`` field are downloaded using the standard Scrapy downloader
|
||||
(which means the downloader middlewares are used, but the spider middlewares
|
||||
aren't). The item remains "locked" at that particular pipeline stage until
|
||||
the files have finished downloading (or failed for some reason).
|
||||
|
||||
4. When the files are downloaded, another field (``files``) will be populated
|
||||
with the results. This field will contain a list of dicts with information
|
||||
|
|
@ -548,10 +547,9 @@ See here the methods that you can override in your custom Files Pipeline:
|
|||
|
||||
.. method:: FilesPipeline.get_media_requests(item, info)
|
||||
|
||||
As seen on the workflow, the pipeline will get the URLs of the images to
|
||||
download from the item. In order to do this, you can override the
|
||||
:meth:`~get_media_requests` method and return a Request for each
|
||||
file URL:
|
||||
As seen on the workflow, the pipeline will get the requests for the files
|
||||
to download from the item by calling this method. You can override it to
|
||||
change what requests are returned:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
@ -591,8 +589,9 @@ See here the methods that you can override in your custom Files Pipeline:
|
|||
* ``downloaded`` - file was downloaded.
|
||||
* ``uptodate`` - file was not downloaded, as it was downloaded recently,
|
||||
according to the file expiration policy.
|
||||
* ``cached`` - file was already scheduled for download, by another item
|
||||
sharing the same file.
|
||||
* ``cached`` - file was taken from a cache (the response has a
|
||||
``"cached"`` flag, e.g. from
|
||||
:class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware`).
|
||||
|
||||
The list of tuples received by :meth:`~item_completed` is
|
||||
guaranteed to retain the same order of the requests returned from the
|
||||
|
|
@ -619,9 +618,6 @@ See here the methods that you can override in your custom Files Pipeline:
|
|||
(False, Failure(...)),
|
||||
]
|
||||
|
||||
By default the :meth:`get_media_requests` method returns ``None`` which
|
||||
means there are no files to download for the item.
|
||||
|
||||
.. method:: FilesPipeline.item_completed(results, item, info)
|
||||
|
||||
The :meth:`FilesPipeline.item_completed` method called when all file
|
||||
|
|
|
|||
|
|
@ -17,8 +17,10 @@ Run Scrapy from a script
|
|||
You can use the :ref:`API <topics-api>` to run Scrapy from a script, instead of
|
||||
the typical way of running Scrapy via ``scrapy crawl``.
|
||||
|
||||
Remember that Scrapy is built on top of the Twisted
|
||||
asynchronous networking library, so you need to run it inside the Twisted reactor.
|
||||
Remember that Scrapy requires a Twisted reactor or (with
|
||||
:setting:`TWISTED_REACTOR_ENABLED` set to ``False``) an asyncio event loop, so
|
||||
you need to run one of those in your script for it to work (helpers described
|
||||
below can do it for you).
|
||||
|
||||
The first utility you can use to run your spiders is
|
||||
:class:`scrapy.crawler.AsyncCrawlerProcess` or
|
||||
|
|
|
|||
|
|
@ -1101,8 +1101,8 @@ Response objects
|
|||
|
||||
The IP address of the server from which the Response originated.
|
||||
|
||||
This attribute is currently only populated by the HTTP 1.1 download
|
||||
handler, i.e. for ``http(s)`` responses. For other handlers,
|
||||
This attribute is currently only populated by the HTTP download
|
||||
handlers, i.e. for ``http(s)`` responses. For other handlers,
|
||||
:attr:`ip_address` is always ``None``.
|
||||
|
||||
.. attribute:: Response.protocol
|
||||
|
|
|
|||
|
|
@ -418,7 +418,9 @@ Default: ``None``
|
|||
|
||||
Import path of a given ``asyncio`` event loop class.
|
||||
|
||||
If the asyncio reactor is enabled (see :setting:`TWISTED_REACTOR`) this setting can be used to specify the
|
||||
If the asyncio reactor is enabled (see :setting:`TWISTED_REACTOR`) or when
|
||||
:ref:`running Scrapy without a reactor <asyncio-without-reactor>` this setting
|
||||
can be used to specify the
|
||||
asyncio event loop to be used with it. Set the setting to the import path of the
|
||||
desired asyncio event loop class. If the setting is set to ``None`` the default asyncio
|
||||
event loop will be used.
|
||||
|
|
@ -554,6 +556,8 @@ performed to any single domain.
|
|||
See also: :ref:`topics-autothrottle` and its
|
||||
:setting:`AUTOTHROTTLE_TARGET_CONCURRENCY` option.
|
||||
|
||||
It is possible to change this setting per domain by using
|
||||
:setting:`DOWNLOAD_SLOTS`.
|
||||
|
||||
.. setting:: DEFAULT_DROPITEM_LOG_LEVEL
|
||||
|
||||
|
|
@ -917,9 +921,8 @@ desired.
|
|||
|
||||
This delay can be set per spider using :attr:`download_delay` spider attribute.
|
||||
|
||||
It is also possible to change this setting per domain, although it requires
|
||||
non-trivial code. See the implementation of the :ref:`AutoThrottle
|
||||
<topics-autothrottle>` extension for an example.
|
||||
It is possible to change this setting per domain by using
|
||||
:setting:`DOWNLOAD_SLOTS`.
|
||||
|
||||
.. setting:: DOWNLOAD_BIND_ADDRESS
|
||||
|
||||
|
|
@ -1358,14 +1361,19 @@ FORCE_CRAWLER_PROCESS
|
|||
Default: ``False``
|
||||
|
||||
If ``False``, :ref:`Scrapy commands that need a CrawlerProcess
|
||||
<topics-commands-crawlerprocess>` will decide between using
|
||||
<topics-commands-crawlerprocess>`, when :setting:`TWISTED_REACTOR_ENABLED`
|
||||
is set to ``True``, will decide between using
|
||||
:class:`scrapy.crawler.AsyncCrawlerProcess` and
|
||||
:class:`scrapy.crawler.CrawlerProcess` based on the value of the
|
||||
:setting:`TWISTED_REACTOR` setting, but ignoring its value in :ref:`per-spider
|
||||
settings <spider-settings>`.
|
||||
|
||||
If ``True``, these commands will always use
|
||||
:class:`~scrapy.crawler.CrawlerProcess`.
|
||||
:class:`~scrapy.crawler.CrawlerProcess` when :setting:`TWISTED_REACTOR_ENABLED`
|
||||
is set to ``True``.
|
||||
|
||||
When :setting:`TWISTED_REACTOR_ENABLED` is set to ``False``,
|
||||
:class:`~scrapy.crawler.AsyncCrawlerProcess` will be used in all cases.
|
||||
|
||||
Set this to ``True`` if you want to set :setting:`TWISTED_REACTOR` to a
|
||||
non-default value in :ref:`per-spider settings <spider-settings>`.
|
||||
|
|
@ -1746,6 +1754,9 @@ The randomization policy is the same used by `wget`_ ``--random-wait`` option.
|
|||
|
||||
If :setting:`DOWNLOAD_DELAY` is zero this option has no effect.
|
||||
|
||||
It is possible to change this setting per domain by using
|
||||
:setting:`DOWNLOAD_SLOTS`.
|
||||
|
||||
.. _wget: https://www.gnu.org/software/wget/manual/wget.html
|
||||
|
||||
.. setting:: REACTOR_THREADPOOL_MAXSIZE
|
||||
|
|
@ -2159,6 +2170,9 @@ Default: ``"twisted.internet.asyncioreactor.AsyncioSelectorReactor"``
|
|||
|
||||
Import path of a given :mod:`~twisted.internet.reactor`.
|
||||
|
||||
.. note::
|
||||
This setting has no effect when :setting:`TWISTED_REACTOR_ENABLED` is ``False``.
|
||||
|
||||
Scrapy will install this reactor if no other reactor is installed yet, such as
|
||||
when the ``scrapy`` CLI program is invoked or when using the
|
||||
:class:`~scrapy.crawler.AsyncCrawlerProcess` class or the
|
||||
|
|
|
|||
|
|
@ -454,7 +454,7 @@ bytes_received
|
|||
.. signal:: bytes_received
|
||||
.. function:: bytes_received(data, request, spider)
|
||||
|
||||
Sent by the HTTP 1.1 and S3 download handlers when a group of bytes is
|
||||
Sent by some download handlers when a group of bytes is
|
||||
received for a specific request. This signal might be fired multiple
|
||||
times for the same request, with partial data each time. For instance,
|
||||
a possible scenario for a 25 kb response would be two signals fired
|
||||
|
|
@ -482,7 +482,7 @@ headers_received
|
|||
.. signal:: headers_received
|
||||
.. function:: headers_received(headers, body_length, request, spider)
|
||||
|
||||
Sent by the HTTP 1.1 and S3 download handlers when the response headers are
|
||||
Sent by some download handlers when the response headers are
|
||||
available for a given request, before downloading any additional content.
|
||||
|
||||
Handlers for this signal can stop the download of a response while it
|
||||
|
|
|
|||
|
|
@ -614,7 +614,7 @@ XMLFeedSpider
|
|||
|
||||
A string which defines the iterator to use. It can be either:
|
||||
|
||||
- ``'iternodes'`` - a fast iterator based on regular expressions
|
||||
- ``'iternodes'`` - a fast iterator based on ``lxml``
|
||||
|
||||
- ``'html'`` - an iterator which uses :class:`~scrapy.Selector`.
|
||||
Keep in mind this uses DOM parsing and must load all DOM in memory
|
||||
|
|
|
|||
|
|
@ -10,8 +10,8 @@ Collector, and can be accessed through the :attr:`~scrapy.crawler.Crawler.stats`
|
|||
attribute of the :ref:`topics-api-crawler`, as illustrated by the examples in
|
||||
the :ref:`topics-stats-usecases` section below.
|
||||
|
||||
However, the Stats Collector is always available, so you can always import it
|
||||
in your module and use its API (to increment or set new stat keys), regardless
|
||||
The Stats Collector API is always available, so you can always use it (to
|
||||
increment or set new stat keys), regardless
|
||||
of whether the stats collection is enabled or not. If it's disabled, the API
|
||||
will still work but it won't collect anything. This is aimed at simplifying the
|
||||
stats collector usage: you should spend no more than one line of code for
|
||||
|
|
@ -21,9 +21,6 @@ using the Stats Collector from.
|
|||
Another feature of the Stats Collector is that it's very efficient (when
|
||||
enabled) and extremely efficient (almost unnoticeable) when disabled.
|
||||
|
||||
The Stats Collector keeps a stats table per open spider which is automatically
|
||||
opened when the spider is opened, and closed when the spider is closed.
|
||||
|
||||
.. _topics-stats-usecases:
|
||||
|
||||
Common Stats Collector uses
|
||||
|
|
@ -87,13 +84,13 @@ Get all stats:
|
|||
Available Stats Collectors
|
||||
==========================
|
||||
|
||||
.. currentmodule:: scrapy.statscollectors
|
||||
|
||||
Besides the basic :class:`StatsCollector` there are other Stats Collectors
|
||||
available in Scrapy which extend the basic Stats Collector. You can select
|
||||
which Stats Collector to use through the :setting:`STATS_CLASS` setting. The
|
||||
default Stats Collector used is the :class:`MemoryStatsCollector`.
|
||||
|
||||
.. currentmodule:: scrapy.statscollectors
|
||||
|
||||
MemoryStatsCollector
|
||||
--------------------
|
||||
|
||||
|
|
@ -102,7 +99,7 @@ MemoryStatsCollector
|
|||
A simple stats collector that keeps the stats of the last scraping run (for
|
||||
each spider) in memory, after they're closed. The stats can be accessed
|
||||
through the :attr:`spider_stats` attribute, which is a dict keyed by spider
|
||||
domain name.
|
||||
name.
|
||||
|
||||
This is the default Stats Collector used in Scrapy.
|
||||
|
||||
|
|
|
|||
|
|
@ -107,8 +107,8 @@ Here are some example tasks you can do with the telnet console:
|
|||
View engine status
|
||||
------------------
|
||||
|
||||
You can use the ``est()`` method of the Scrapy engine to quickly show its state
|
||||
using the telnet console::
|
||||
You can use the ``est()`` method provided by the console to quickly show the
|
||||
engine status::
|
||||
|
||||
telnet localhost 6023
|
||||
>>> est()
|
||||
|
|
|
|||
|
|
@ -289,11 +289,11 @@ class Scheduler(BaseScheduler):
|
|||
|
||||
:param dqclass: A class to be used as persistent request queue.
|
||||
The value for the :setting:`SCHEDULER_DISK_QUEUE` setting is used by default.
|
||||
:type dqclass: class
|
||||
:type dqclass: type
|
||||
|
||||
:param mqclass: A class to be used as non-persistent request queue.
|
||||
The value for the :setting:`SCHEDULER_MEMORY_QUEUE` setting is used by default.
|
||||
:type mqclass: class
|
||||
:type mqclass: type
|
||||
|
||||
:param logunser: A boolean that indicates whether or not unserializable requests should be logged.
|
||||
The value for the :setting:`SCHEDULER_DEBUG` setting is used by default.
|
||||
|
|
@ -306,7 +306,7 @@ class Scheduler(BaseScheduler):
|
|||
|
||||
:param pqclass: A class to be used as priority queue for requests.
|
||||
The value for the :setting:`SCHEDULER_PRIORITY_QUEUE` setting is used by default.
|
||||
:type pqclass: class
|
||||
:type pqclass: type
|
||||
|
||||
:param crawler: The crawler object corresponding to the current crawl.
|
||||
:type crawler: :class:`scrapy.crawler.Crawler`
|
||||
|
|
|
|||
|
|
@ -288,10 +288,10 @@ class BaseSettings(MutableMapping[str, Any]):
|
|||
- ``['one', 'two']`` if set to ``'["one", "two"]'`` or ``'one,two'``
|
||||
|
||||
:param name: the setting name
|
||||
:type name: string
|
||||
:type name: str
|
||||
|
||||
:param default: the value to return if no setting is found
|
||||
:type default: any
|
||||
:type default: object
|
||||
"""
|
||||
value = self.get(name, default)
|
||||
if value is None:
|
||||
|
|
|
|||
Loading…
Reference in New Issue