mirror of https://github.com/scrapy/scrapy.git
Assorted docs fixes, part 1. (#7710)
This commit is contained in:
Andrey Rakhmatullin
GitHub
parent
c5ec881f1d
commit
bdf3067935
|
|
@ -1,68 +0,0 @@
|
|||
:orphan:
|
||||
|
||||
======================================
|
||||
Scrapy documentation quick start guide
|
||||
======================================
|
||||
|
||||
This file provides a quick guide on how to compile the Scrapy documentation.
|
||||
|
||||
|
||||
Setup the environment
|
||||
---------------------
|
||||
|
||||
To compile the documentation you need Sphinx Python library. To install it
|
||||
and all its dependencies run the following command from this dir
|
||||
|
||||
::
|
||||
|
||||
pip install -r requirements.txt
|
||||
|
||||
|
||||
Compile the documentation
|
||||
-------------------------
|
||||
|
||||
To compile the documentation (to classic HTML output) run the following command
|
||||
from this dir::
|
||||
|
||||
make html
|
||||
|
||||
Documentation will be generated (in HTML format) inside the ``build/html`` dir.
|
||||
|
||||
|
||||
View the documentation
|
||||
----------------------
|
||||
|
||||
To view the documentation run the following command::
|
||||
|
||||
make htmlview
|
||||
|
||||
This command will fire up your default browser and open the main page of your
|
||||
(previously generated) HTML documentation.
|
||||
|
||||
|
||||
Start over
|
||||
----------
|
||||
|
||||
To clean up all generated documentation files and start from scratch run::
|
||||
|
||||
make clean
|
||||
|
||||
Keep in mind that this command won't touch any documentation source files.
|
||||
|
||||
|
||||
Recreating documentation on the fly
|
||||
-----------------------------------
|
||||
|
||||
There is a way to recreate the doc automatically when you make changes, you
|
||||
need to install watchdog (``pip install watchdog``) and then use::
|
||||
|
||||
make watch
|
||||
|
||||
Alternative method using tox
|
||||
----------------------------
|
||||
|
||||
To compile the documentation to HTML run the following command::
|
||||
|
||||
tox -e docs
|
||||
|
||||
Documentation will be generated inside the ``docs/_build/all`` dir.
|
||||
|
|
@ -323,9 +323,10 @@ deprecation removals are documented in the :ref:`release notes <news>`.
|
|||
Tests
|
||||
=====
|
||||
|
||||
Tests are implemented using the :doc:`Twisted unit-testing framework
|
||||
<twisted:development/test-standard>`. Running tests requires
|
||||
:doc:`tox <tox:index>`.
|
||||
Tests are implemented using pytest_. Running tests requires :doc:`tox
|
||||
<tox:index>`.
|
||||
|
||||
.. _pytest: https://pytest.org
|
||||
|
||||
.. _running-tests:
|
||||
|
||||
|
|
|
|||
|
|
@ -285,7 +285,8 @@ consume a lot of memory.
|
|||
In order to avoid parsing all the entire feed at once in memory, you can use
|
||||
the :func:`~scrapy.utils.iterators.xmliter_lxml` and
|
||||
:func:`~scrapy.utils.iterators.csviter` functions. In fact, this is what
|
||||
:class:`~scrapy.spiders.XMLFeedSpider` uses.
|
||||
:class:`~scrapy.spiders.XMLFeedSpider` and
|
||||
:class:`~scrapy.spiders.CSVFeedSpider` use.
|
||||
|
||||
.. autofunction:: scrapy.utils.iterators.xmliter_lxml
|
||||
|
||||
|
|
@ -360,10 +361,11 @@ method for this purpose. For example:
|
|||
def process_spider_output(self, response, result):
|
||||
for item_or_request in result:
|
||||
if isinstance(item_or_request, Request):
|
||||
yield item_or_request
|
||||
continue
|
||||
adapter = ItemAdapter(item)
|
||||
adapter = ItemAdapter(item_or_request)
|
||||
for _ in range(adapter["multiply_by"]):
|
||||
yield deepcopy(item)
|
||||
yield deepcopy(item_or_request)
|
||||
|
||||
Does Scrapy support IPv6 addresses?
|
||||
-----------------------------------
|
||||
|
|
|
|||
|
|
@ -24,7 +24,7 @@ Having trouble? We'd like to help!
|
|||
* Ask or search questions in `StackOverflow using the scrapy tag`_.
|
||||
* Ask or search questions in the `Scrapy subreddit`_.
|
||||
* Search for questions on the archives of the `scrapy-users mailing list`_.
|
||||
* Ask a question in the `#scrapy IRC channel`_,
|
||||
* Ask a question in the `#scrapy IRC channel`_.
|
||||
* Report bugs with Scrapy in our `issue tracker`_.
|
||||
* Join the Discord community `Scrapy Discord`_.
|
||||
|
||||
|
|
@ -91,15 +91,15 @@ Basic concepts
|
|||
:doc:`topics/selectors`
|
||||
Extract the data from web pages using XPath.
|
||||
|
||||
:doc:`topics/shell`
|
||||
Test your extraction code in an interactive environment.
|
||||
|
||||
:doc:`topics/items`
|
||||
Define the data you want to scrape.
|
||||
|
||||
:doc:`topics/loaders`
|
||||
Populate your items with the extracted data.
|
||||
|
||||
:doc:`topics/shell`
|
||||
Test your extraction code in an interactive environment.
|
||||
|
||||
:doc:`topics/item-pipeline`
|
||||
Post-process and store your scraped data.
|
||||
|
||||
|
|
|
|||
|
|
@ -90,9 +90,10 @@ to figure these settings out automatically.
|
|||
.. note::
|
||||
|
||||
This is using :ref:`feed exports <topics-feed-exports>` to generate the
|
||||
JSON file, you can easily change the export format (XML or CSV, for example) or the
|
||||
storage backend (FTP or `Amazon S3`_, for example). You can also write an
|
||||
:ref:`item pipeline <topics-item-pipeline>` to store the items in a database.
|
||||
JSON Lines file, you can easily change the export format (XML or CSV, for
|
||||
example) or the storage backend (FTP or `Amazon S3`_, for example). You can
|
||||
also write an :ref:`item pipeline <topics-item-pipeline>` to store the
|
||||
items in a database.
|
||||
|
||||
|
||||
.. _topics-whatelse:
|
||||
|
|
|
|||
|
|
@ -1035,7 +1035,7 @@ Deprecations
|
|||
New features
|
||||
~~~~~~~~~~~~
|
||||
|
||||
- Added a new setting, :setting:`REFERER_POLICIES`, to allow customizing
|
||||
- Added a new setting, :setting:`REFERRER_POLICIES`, to allow customizing
|
||||
supported referrer policies.
|
||||
|
||||
Bug fixes
|
||||
|
|
@ -6046,7 +6046,7 @@ The following changes may impact custom priority queue classes:
|
|||
|
||||
* A new keyword parameter has been added: ``key``. It is a string
|
||||
that is always an empty string for memory queues and indicates the
|
||||
:setting:`JOB_DIR` value for disk queues.
|
||||
:setting:`JOBDIR` value for disk queues.
|
||||
|
||||
* The parameter for disk queues that contains data from the previous
|
||||
crawl, ``startprios`` or ``slot_startprios``, is now passed as a
|
||||
|
|
@ -6600,7 +6600,7 @@ New features
|
|||
``scrapy.pqueues.DownloaderAwarePriorityQueue``, may be
|
||||
:ref:`enabled <broad-crawls-scheduler-priority-queue>` for a significant
|
||||
scheduling improvement on crawls targeting multiple web domains, at the
|
||||
cost of no :setting:`CONCURRENT_REQUESTS_PER_IP` support (:issue:`3520`)
|
||||
cost of no ``CONCURRENT_REQUESTS_PER_IP`` support (:issue:`3520`)
|
||||
|
||||
* A new :attr:`.Request.cb_kwargs` attribute
|
||||
provides a cleaner way to pass keyword arguments to callback methods
|
||||
|
|
@ -9079,7 +9079,7 @@ New features and settings
|
|||
- In request errbacks, offending requests are now received in ``failure.request`` attribute (:rev:`2738`)
|
||||
- Big downloader refactoring to support per domain/ip concurrency limits (:rev:`2732`)
|
||||
- ``CONCURRENT_REQUESTS_PER_SPIDER`` setting has been deprecated and replaced by:
|
||||
- :setting:`CONCURRENT_REQUESTS`, :setting:`CONCURRENT_REQUESTS_PER_DOMAIN`, :setting:`CONCURRENT_REQUESTS_PER_IP`
|
||||
- :setting:`CONCURRENT_REQUESTS`, :setting:`CONCURRENT_REQUESTS_PER_DOMAIN`, ``CONCURRENT_REQUESTS_PER_IP``
|
||||
- check the documentation for more details
|
||||
- Added builtin caching DNS resolver (:rev:`2728`)
|
||||
- Moved Amazon AWS-related components/extensions (SQS spider queue, SimpleDB stats collector) to a separate project: [scaws](https://github.com/scrapinghub/scaws) (:rev:`2706`, :rev:`2714`)
|
||||
|
|
|
|||
|
|
@ -105,6 +105,9 @@ Scrapy API requires passing a Deferred to it) using the following helpers:
|
|||
|
||||
.. autofunction:: scrapy.utils.defer.deferred_from_coro
|
||||
.. autofunction:: scrapy.utils.defer.deferred_f_from_coro_f
|
||||
|
||||
The following function helps with a reverse wrapping:
|
||||
|
||||
.. autofunction:: scrapy.utils.defer.ensure_awaitable
|
||||
|
||||
|
||||
|
|
@ -190,7 +193,7 @@ in future Scrapy versions. The following features are not available:
|
|||
:class:`~scrapy.crawler.CrawlerProcess`
|
||||
(:class:`~scrapy.crawler.AsyncCrawlerProcess` and
|
||||
:class:`~scrapy.crawler.AsyncCrawlerRunner` are available)
|
||||
* Twisted-specific DNS resolvers (the :setting:`DNS_RESOLVER` setting)
|
||||
* Twisted-specific DNS resolvers (the :setting:`TWISTED_DNS_RESOLVER` setting)
|
||||
* User and 3rd-party code that requires a reactor (see :ref:`below
|
||||
<asyncio-without-reactor-migrate>` for examples)
|
||||
|
||||
|
|
@ -315,8 +318,7 @@ implementations, :class:`~asyncio.ProactorEventLoop` (default) and
|
|||
:class:`~asyncio.SelectorEventLoop` works with Twisted.
|
||||
|
||||
Scrapy changes the event loop class to :class:`~asyncio.SelectorEventLoop`
|
||||
automatically when you change the :setting:`TWISTED_REACTOR` setting or call
|
||||
:func:`~scrapy.utils.reactor.install_reactor`.
|
||||
automatically when installing the asyncio reactor.
|
||||
|
||||
.. note:: Other libraries you use may require
|
||||
:class:`~asyncio.ProactorEventLoop`, e.g. because it supports
|
||||
|
|
|
|||
|
|
@ -199,6 +199,7 @@ Global commands:
|
|||
* :command:`fetch`
|
||||
* :command:`view`
|
||||
* :command:`version`
|
||||
* :command:`bench`
|
||||
|
||||
Project-only commands:
|
||||
|
||||
|
|
@ -207,7 +208,6 @@ Project-only commands:
|
|||
* :command:`list`
|
||||
* :command:`edit`
|
||||
* :command:`parse`
|
||||
* :command:`bench`
|
||||
|
||||
.. command:: startproject
|
||||
|
||||
|
|
@ -476,7 +476,7 @@ Supported options:
|
|||
|
||||
* ``--spider=SPIDER``: bypass spider autodetection and force use of specific spider
|
||||
|
||||
* ``--a NAME=VALUE``: set spider argument (may be repeated)
|
||||
* ``-a NAME=VALUE``: set spider argument (may be repeated)
|
||||
|
||||
* ``--callback`` or ``-c``: spider method to use as callback for parsing the
|
||||
response
|
||||
|
|
|
|||
|
|
@ -268,5 +268,5 @@ You can also send multiple requests in parallel:
|
|||
yield {
|
||||
"h1": response.css("h1::text").get(),
|
||||
"price": responses[0].css(".price::text").get(),
|
||||
"price2": responses[1].css(".color::text").get(),
|
||||
"color": responses[1].css(".color::text").get(),
|
||||
}
|
||||
|
|
|
|||
|
|
@ -505,7 +505,7 @@ Filesystem storage backend (default)
|
|||
|
||||
* ``response_body`` - the plain response body
|
||||
|
||||
* ``response_headers`` - the request headers (in raw HTTP format)
|
||||
* ``response_headers`` - the response headers (in raw HTTP format)
|
||||
|
||||
* ``meta`` - some metadata of this cache resource in Python ``repr()``
|
||||
format (grep-friendly format)
|
||||
|
|
@ -547,7 +547,7 @@ defines the methods described below.
|
|||
.. method:: open_spider(spider)
|
||||
|
||||
This method gets called after a spider has been opened for crawling. It handles
|
||||
the :signal:`open_spider <spider_opened>` signal.
|
||||
the :signal:`spider_opened` signal.
|
||||
|
||||
:param spider: the spider which has been opened
|
||||
:type spider: :class:`~scrapy.Spider` object
|
||||
|
|
@ -555,7 +555,7 @@ defines the methods described below.
|
|||
.. method:: close_spider(spider)
|
||||
|
||||
This method gets called after a spider has been closed. It handles
|
||||
the :signal:`close_spider <spider_closed>` signal.
|
||||
the :signal:`spider_closed` signal.
|
||||
|
||||
:param spider: the spider which has been closed
|
||||
:type spider: :class:`~scrapy.Spider` object
|
||||
|
|
@ -814,7 +814,6 @@ HttpProxyMiddleware settings
|
|||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. setting:: HTTPPROXY_ENABLED
|
||||
.. setting:: HTTPPROXY_AUTH_ENCODING
|
||||
|
||||
HTTPPROXY_ENABLED
|
||||
^^^^^^^^^^^^^^^^^
|
||||
|
|
@ -823,6 +822,8 @@ Default: ``True``
|
|||
|
||||
Whether or not to enable the :class:`HttpProxyMiddleware`.
|
||||
|
||||
.. setting:: HTTPPROXY_AUTH_ENCODING
|
||||
|
||||
HTTPPROXY_AUTH_ENCODING
|
||||
^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
|
|
@ -984,7 +985,7 @@ Whether the Meta Refresh middleware will be enabled.
|
|||
METAREFRESH_IGNORE_TAGS
|
||||
^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Default: ``[]``
|
||||
Default: ``["noscript"]``
|
||||
|
||||
Meta tags within these tags are ignored.
|
||||
|
||||
|
|
@ -1091,7 +1092,7 @@ Default::
|
|||
'twisted.internet.error.ConnectionDone',
|
||||
'twisted.internet.error.ConnectError',
|
||||
'twisted.internet.error.ConnectionLost',
|
||||
IOError,
|
||||
OSError,
|
||||
'scrapy.core.downloader.handlers.http11.TunnelError',
|
||||
]
|
||||
|
||||
|
|
|
|||
|
|
@ -133,7 +133,7 @@ data from it depends on the type of response:
|
|||
|
||||
.. code-block:: python
|
||||
|
||||
selector = Selector(data["html"])
|
||||
selector = Selector(text=data["html"])
|
||||
|
||||
- If the response is JavaScript, or HTML with a ``<script/>`` element
|
||||
containing the desired data, see :ref:`topics-parsing-javascript`.
|
||||
|
|
|
|||
|
|
@ -31,7 +31,7 @@ For example:
|
|||
.. code-block:: python
|
||||
|
||||
def parse_page(self, response):
|
||||
if "Bandwidth exceeded" in response.body:
|
||||
if "Bandwidth exceeded" in response.text:
|
||||
raise CloseSpider("bandwidth_exceeded")
|
||||
|
||||
DontCloseSpider
|
||||
|
|
|
|||
|
|
@ -153,7 +153,7 @@ output examples, which assume you're exporting these two items:
|
|||
BaseItemExporter
|
||||
----------------
|
||||
|
||||
.. class:: BaseItemExporter(fields_to_export=None, export_empty_fields=False, encoding='utf-8', indent=0, dont_fail=False)
|
||||
.. class:: BaseItemExporter(fields_to_export=None, export_empty_fields=False, encoding=None, indent=None, dont_fail=False)
|
||||
|
||||
This is the (abstract) base class for all Item Exporters. It provides
|
||||
support for common features used by all (concrete) Item Exporters, such as
|
||||
|
|
@ -239,7 +239,7 @@ BaseItemExporter
|
|||
|
||||
.. attribute:: indent
|
||||
|
||||
Amount of spaces used to indent the output on each level. Defaults to ``0``.
|
||||
Amount of spaces used to indent the output on each level. Defaults to ``None``.
|
||||
|
||||
* ``indent=None`` selects the most compact representation,
|
||||
all items in the same line with no indentation
|
||||
|
|
@ -328,7 +328,7 @@ CsvItemExporter
|
|||
|
||||
:param join_multivalued: The char (or chars) that will be used for joining
|
||||
multi-valued fields, if found.
|
||||
:type include_headers_line: str
|
||||
:type join_multivalued: str
|
||||
|
||||
:param errors: The optional string that specifies how encoding and decoding
|
||||
errors are to be handled. For more information see
|
||||
|
|
@ -342,14 +342,14 @@ CsvItemExporter
|
|||
|
||||
A typical output of this exporter would be::
|
||||
|
||||
product,price
|
||||
name,price
|
||||
Color TV,1200
|
||||
DVD player,200
|
||||
|
||||
PickleItemExporter
|
||||
------------------
|
||||
|
||||
.. class:: PickleItemExporter(file, protocol=0, **kwargs)
|
||||
.. class:: PickleItemExporter(file, protocol=4, **kwargs)
|
||||
|
||||
Exports items in pickle format to the given file-like object.
|
||||
|
||||
|
|
|
|||
|
|
@ -341,9 +341,6 @@ closing the spider. If the spider generates more than that number of errors,
|
|||
it will be closed with the reason ``closespider_errorcount``. If zero (or non
|
||||
set), spiders won't be closed by number of errors.
|
||||
|
||||
.. module:: scrapy.extensions.debug
|
||||
:synopsis: Extensions for debugging Scrapy
|
||||
|
||||
.. module:: scrapy.extensions.periodic_log
|
||||
:synopsis: Periodic stats logging
|
||||
|
||||
|
|
@ -418,7 +415,7 @@ Example extension configuration:
|
|||
custom_settings = {
|
||||
"LOG_LEVEL": "INFO",
|
||||
"PERIODIC_LOG_STATS": {
|
||||
"include": ["downloader/", "scheduler/", "log_count/", "item_scraped_count/"],
|
||||
"include": ["downloader/", "scheduler/", "log_count/", "item_scraped_count"],
|
||||
},
|
||||
"PERIODIC_LOG_DELTA": {"include": ["downloader/"]},
|
||||
"PERIODIC_LOG_TIMING_ENABLED": True,
|
||||
|
|
@ -463,6 +460,9 @@ Default: ``False``
|
|||
Debugging extensions
|
||||
--------------------
|
||||
|
||||
.. module:: scrapy.extensions.debug
|
||||
:synopsis: Extensions for debugging Scrapy
|
||||
|
||||
Stack trace dump extension
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
|
|
|
|||
|
|
@ -615,6 +615,7 @@ Default:
|
|||
"file": "scrapy.extensions.feedexport.FileFeedStorage",
|
||||
"stdout": "scrapy.extensions.feedexport.StdoutFeedStorage",
|
||||
"s3": "scrapy.extensions.feedexport.S3FeedStorage",
|
||||
"gs": "scrapy.extensions.feedexport.GCSFeedStorage",
|
||||
"ftp": "scrapy.extensions.feedexport.FTPFeedStorage",
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -376,7 +376,9 @@ Creating dicts from items:
|
|||
>>> dict(product) # create a dict from all populated values
|
||||
{'price': 1000, 'name': 'Desktop PC'}
|
||||
|
||||
Creating items from dicts:
|
||||
Creating items from dicts:
|
||||
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> Product({"name": "Laptop PC", "price": 1500})
|
||||
Product(price=1500, name='Laptop PC')
|
||||
|
|
|
|||
|
|
@ -93,7 +93,7 @@ You can get the oldest object of each class using the
|
|||
Which objects are tracked?
|
||||
--------------------------
|
||||
|
||||
The objects tracked by ``trackrefs`` are all from these classes (and all its
|
||||
The objects tracked by ``trackref`` are all from these classes (and all its
|
||||
subclasses):
|
||||
|
||||
* :class:`scrapy.Request`
|
||||
|
|
@ -187,7 +187,7 @@ Here are the functions available in the :mod:`~scrapy.utils.trackref` module.
|
|||
Inherit from this class if you want to track live
|
||||
instances with the ``trackref`` module.
|
||||
|
||||
.. function:: print_live_refs(class_name, ignore=NoneType)
|
||||
.. function:: print_live_refs(ignore=NoneType)
|
||||
|
||||
Print a report of live references, grouped by class name.
|
||||
|
||||
|
|
@ -203,9 +203,9 @@ Here are the functions available in the :mod:`~scrapy.utils.trackref` module.
|
|||
|
||||
.. function:: iter_all(class_name)
|
||||
|
||||
Return an iterator over all objects alive with the given class name, or
|
||||
``None`` if none is found. Use :func:`print_live_refs` first to get a list
|
||||
of all tracked live objects per class name.
|
||||
Return an iterator over all objects alive with the given class name. Use
|
||||
:func:`print_live_refs` first to get a list of all tracked live objects
|
||||
per class name.
|
||||
|
||||
.. skip: end
|
||||
|
||||
|
|
|
|||
|
|
@ -76,7 +76,7 @@ data that will be assigned to the ``name`` field later.
|
|||
|
||||
Afterwards, similar calls are used for ``price`` and ``stock`` fields
|
||||
(the latter using a CSS selector with the :meth:`~ItemLoader.add_css` method),
|
||||
and finally the ``last_update`` field is populated directly with a literal value
|
||||
and finally the ``last_updated`` field is populated directly with a literal value
|
||||
(``today``) using a different method: :meth:`~ItemLoader.add_value`.
|
||||
|
||||
Finally, when all data is collected, the :meth:`ItemLoader.load_item` method is
|
||||
|
|
@ -273,8 +273,8 @@ The precedence order, for both input and output processors, is as follows:
|
|||
1. Item Loader field-specific attributes: ``field_in`` and ``field_out`` (most
|
||||
precedence)
|
||||
2. Field metadata (``input_processor`` and ``output_processor`` key)
|
||||
3. Item Loader defaults: :meth:`ItemLoader.default_input_processor` and
|
||||
:meth:`ItemLoader.default_output_processor` (least precedence)
|
||||
3. Item Loader defaults: :attr:`ItemLoader.default_input_processor` and
|
||||
:attr:`ItemLoader.default_output_processor` (least precedence)
|
||||
|
||||
See also: :ref:`topics-loaders-extending`.
|
||||
|
||||
|
|
@ -323,8 +323,8 @@ There are several ways to modify Item Loader context values:
|
|||
loader = ItemLoader(product, unit="cm")
|
||||
|
||||
3. On Item Loader declaration, for those input/output processors that support
|
||||
instantiating them with an Item Loader context. :class:`~processor.MapCompose` is one of
|
||||
them:
|
||||
instantiating them with an Item Loader context.
|
||||
:class:`~itemloaders.processors.MapCompose` is one of them:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
|
|||
|
|
@ -4,11 +4,6 @@
|
|||
Logging
|
||||
=======
|
||||
|
||||
.. note::
|
||||
:mod:`scrapy.log` has been deprecated alongside its functions in favor of
|
||||
explicit calls to the Python standard logging. Keep reading to learn more
|
||||
about the new logging system.
|
||||
|
||||
Scrapy uses :mod:`logging` for event logging. We'll
|
||||
provide some simple examples to get you started, but for more advanced
|
||||
use-cases it's strongly suggested to read thoroughly its documentation.
|
||||
|
|
|
|||
|
|
@ -371,11 +371,12 @@ For the Images Pipeline, set :setting:`IMAGES_URLS_FIELD` and/or
|
|||
If you need something more complex and want to override the custom pipeline
|
||||
behaviour, see :ref:`topics-media-pipeline-override`.
|
||||
|
||||
If you have multiple image pipelines inheriting from ImagePipeline and you want
|
||||
to have different settings in different pipelines you can set setting keys
|
||||
preceded with uppercase name of your pipeline class. E.g. if your pipeline is
|
||||
called MyPipeline and you want to have custom IMAGES_URLS_FIELD you define
|
||||
setting MYPIPELINE_IMAGES_URLS_FIELD and your custom settings will be used.
|
||||
If you have multiple image pipelines inheriting from :class:`ImagesPipeline`
|
||||
and you want to have different settings in different pipelines you can set
|
||||
setting keys preceded with uppercase name of your pipeline class. E.g. if your
|
||||
pipeline is called ``MyPipeline`` and you want to have custom
|
||||
:setting:`IMAGES_URLS_FIELD` you define setting
|
||||
``MYPIPELINE_IMAGES_URLS_FIELD`` and your custom settings will be used.
|
||||
|
||||
|
||||
Additional features
|
||||
|
|
|
|||
|
|
@ -247,7 +247,7 @@ Request objects
|
|||
Return a new Request which is a copy of this Request. See also:
|
||||
:ref:`topics-request-response-ref-request-callback-arguments`.
|
||||
|
||||
.. method:: Request.replace([url, method, headers, body, cookies, meta, flags, encoding, priority, dont_filter, callback, errback, cb_kwargs])
|
||||
.. method:: Request.replace([url, method, headers, body, cookies, meta, flags, encoding, priority, dont_filter, callback, errback, cb_kwargs, cls])
|
||||
|
||||
Return a Request object with the same members, except for those members
|
||||
given new values by whichever keyword arguments are specified. The
|
||||
|
|
@ -717,6 +717,7 @@ Those are:
|
|||
* :reqmeta:`download_fail_on_dataloss`
|
||||
* :reqmeta:`download_latency`
|
||||
* :reqmeta:`download_maxsize`
|
||||
* :reqmeta:`download_slot`
|
||||
* :reqmeta:`download_warnsize`
|
||||
* :reqmeta:`download_timeout`
|
||||
* ``ftp_password`` (See :setting:`FTP_PASSWORD` for more info)
|
||||
|
|
@ -1085,7 +1086,7 @@ Response objects
|
|||
.. attribute:: Response.flags
|
||||
|
||||
A list that contains flags for this response. Flags are labels used for
|
||||
tagging Responses. For example: ``'cached'``, ``'redirected``', etc. And
|
||||
tagging Responses. For example: ``'cached'``, ``'redirected'``', etc. And
|
||||
they're shown on the string representation of the Response (``__str__()``
|
||||
method) which is used by the engine for logging.
|
||||
|
||||
|
|
@ -1119,7 +1120,7 @@ Response objects
|
|||
|
||||
Returns a new Response which is a copy of this Response.
|
||||
|
||||
.. method:: Response.replace([url, status, headers, body, request, flags, cls])
|
||||
.. method:: Response.replace([url, status, headers, body, request, flags, certificate, ip_address, protocol, cls])
|
||||
|
||||
Returns a Response object with the same members, except for those members
|
||||
given new values by whichever keyword arguments are specified. The
|
||||
|
|
|
|||
|
|
@ -308,7 +308,7 @@ Examples:
|
|||
|
||||
* ``*::text`` selects all descendant text nodes of the current selector context:
|
||||
|
||||
..skip: next
|
||||
.. skip: next
|
||||
.. code-block:: pycon
|
||||
|
||||
>>> response.css("#images *::text").getall()
|
||||
|
|
|
|||
|
|
@ -409,77 +409,6 @@ Default: ``{}``
|
|||
A dict containing paths to the add-ons enabled in your project and their
|
||||
priorities. For more information, see :ref:`topics-addons`.
|
||||
|
||||
.. setting:: AWS_ACCESS_KEY_ID
|
||||
|
||||
AWS_ACCESS_KEY_ID
|
||||
-----------------
|
||||
|
||||
Default: ``None``
|
||||
|
||||
The AWS access key used by code that requires access to `Amazon Web services`_,
|
||||
such as the :ref:`S3 feed storage backend <topics-feed-storage-s3>`.
|
||||
|
||||
.. setting:: AWS_SECRET_ACCESS_KEY
|
||||
|
||||
AWS_SECRET_ACCESS_KEY
|
||||
---------------------
|
||||
|
||||
Default: ``None``
|
||||
|
||||
The AWS secret key used by code that requires access to `Amazon Web services`_,
|
||||
such as the :ref:`S3 feed storage backend <topics-feed-storage-s3>`.
|
||||
|
||||
.. setting:: AWS_SESSION_TOKEN
|
||||
|
||||
AWS_SESSION_TOKEN
|
||||
-----------------
|
||||
|
||||
Default: ``None``
|
||||
|
||||
The AWS security token used by code that requires access to `Amazon Web services`_,
|
||||
such as the :ref:`S3 feed storage backend <topics-feed-storage-s3>`, when using
|
||||
`temporary security credentials`_.
|
||||
|
||||
.. _temporary security credentials: https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html
|
||||
|
||||
.. setting:: AWS_ENDPOINT_URL
|
||||
|
||||
AWS_ENDPOINT_URL
|
||||
----------------
|
||||
|
||||
Default: ``None``
|
||||
|
||||
Endpoint URL used for S3-like storage, for example Minio or s3.scality.
|
||||
|
||||
.. setting:: AWS_USE_SSL
|
||||
|
||||
AWS_USE_SSL
|
||||
-----------
|
||||
|
||||
Default: ``None``
|
||||
|
||||
Use this option if you want to disable SSL connection for communication with
|
||||
S3 or S3-like storage. By default SSL will be used.
|
||||
|
||||
.. setting:: AWS_VERIFY
|
||||
|
||||
AWS_VERIFY
|
||||
----------
|
||||
|
||||
Default: ``None``
|
||||
|
||||
Verify SSL connection between Scrapy and S3 or S3-like storage. By default
|
||||
SSL verification will occur.
|
||||
|
||||
.. setting:: AWS_REGION_NAME
|
||||
|
||||
AWS_REGION_NAME
|
||||
---------------
|
||||
|
||||
Default: ``None``
|
||||
|
||||
The name of the region associated with the AWS client.
|
||||
|
||||
.. setting:: ASYNCIO_EVENT_LOOP
|
||||
|
||||
ASYNCIO_EVENT_LOOP
|
||||
|
|
@ -508,6 +437,77 @@ Note that the event loop class must inherit from :class:`asyncio.AbstractEventLo
|
|||
|
||||
.. note:: This is a :ref:`reactor setting <reactor-settings>`.
|
||||
|
||||
.. setting:: AWS_ACCESS_KEY_ID
|
||||
|
||||
AWS_ACCESS_KEY_ID
|
||||
-----------------
|
||||
|
||||
Default: ``None``
|
||||
|
||||
The AWS access key used by code that requires access to `Amazon Web services`_,
|
||||
such as the :ref:`S3 feed storage backend <topics-feed-storage-s3>`.
|
||||
|
||||
.. setting:: AWS_ENDPOINT_URL
|
||||
|
||||
AWS_ENDPOINT_URL
|
||||
----------------
|
||||
|
||||
Default: ``None``
|
||||
|
||||
Endpoint URL used for S3-like storage, for example Minio or s3.scality.
|
||||
|
||||
.. setting:: AWS_REGION_NAME
|
||||
|
||||
AWS_REGION_NAME
|
||||
---------------
|
||||
|
||||
Default: ``None``
|
||||
|
||||
The name of the region associated with the AWS client.
|
||||
|
||||
.. setting:: AWS_SECRET_ACCESS_KEY
|
||||
|
||||
AWS_SECRET_ACCESS_KEY
|
||||
---------------------
|
||||
|
||||
Default: ``None``
|
||||
|
||||
The AWS secret key used by code that requires access to `Amazon Web services`_,
|
||||
such as the :ref:`S3 feed storage backend <topics-feed-storage-s3>`.
|
||||
|
||||
.. setting:: AWS_SESSION_TOKEN
|
||||
|
||||
AWS_SESSION_TOKEN
|
||||
-----------------
|
||||
|
||||
Default: ``None``
|
||||
|
||||
The AWS security token used by code that requires access to `Amazon Web services`_,
|
||||
such as the :ref:`S3 feed storage backend <topics-feed-storage-s3>`, when using
|
||||
`temporary security credentials`_.
|
||||
|
||||
.. _temporary security credentials: https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html
|
||||
|
||||
.. setting:: AWS_USE_SSL
|
||||
|
||||
AWS_USE_SSL
|
||||
-----------
|
||||
|
||||
Default: ``None``
|
||||
|
||||
Use this option if you want to disable SSL connection for communication with
|
||||
S3 or S3-like storage. By default SSL will be used.
|
||||
|
||||
.. setting:: AWS_VERIFY
|
||||
|
||||
AWS_VERIFY
|
||||
----------
|
||||
|
||||
Default: ``None``
|
||||
|
||||
Verify SSL connection between Scrapy and S3 or S3-like storage. By default
|
||||
SSL verification will occur.
|
||||
|
||||
.. setting:: BOT_NAME
|
||||
|
||||
BOT_NAME
|
||||
|
|
@ -593,7 +593,7 @@ When writing an item pipeline, you can force a different log level by setting
|
|||
DEFAULT_ITEM_CLASS
|
||||
------------------
|
||||
|
||||
Default: ``'scrapy.Item'``
|
||||
Default: ``'scrapy.item.Item'``
|
||||
|
||||
The default class that will be used for instantiating items in the :ref:`the
|
||||
Scrapy shell <topics-shell>`.
|
||||
|
|
@ -687,7 +687,7 @@ Whether to enable DNS in-memory cache.
|
|||
:class:`~scrapy.resolver.CachingThreadedResolver` and
|
||||
:class:`~scrapy.resolver.CachingHostnameResolver`. It has no effect when
|
||||
:setting:`TWISTED_REACTOR_ENABLED` is ``False``, and may have no effect
|
||||
either when :setting:`DNS_RESOLVER` is set to a different resolver.
|
||||
either when :setting:`TWISTED_DNS_RESOLVER` is set to a different resolver.
|
||||
|
||||
.. note:: This is a :ref:`reactor setting <reactor-settings>`.
|
||||
|
||||
|
|
@ -702,25 +702,6 @@ DNS in-memory cache size, see :setting:`DNSCACHE_ENABLED`.
|
|||
|
||||
.. note:: This is a :ref:`reactor setting <reactor-settings>`.
|
||||
|
||||
.. setting:: TWISTED_DNS_RESOLVER
|
||||
|
||||
TWISTED_DNS_RESOLVER
|
||||
--------------------
|
||||
|
||||
Default: ``'scrapy.resolver.CachingThreadedResolver'``
|
||||
|
||||
The class to be used by Twisted to resolve DNS names. The default
|
||||
``scrapy.resolver.CachingThreadedResolver`` supports specifying a timeout for
|
||||
DNS requests via the :setting:`DNS_TIMEOUT` setting, but works only with IPv4
|
||||
addresses. Scrapy provides an alternative resolver,
|
||||
``scrapy.resolver.CachingHostnameResolver``, which supports IPv4/IPv6 addresses but does not
|
||||
take the :setting:`DNS_TIMEOUT` setting into account.
|
||||
|
||||
.. note::
|
||||
This setting has no effect when :setting:`TWISTED_REACTOR_ENABLED` is ``False``.
|
||||
|
||||
.. note:: This is a :ref:`reactor setting <reactor-settings>`.
|
||||
|
||||
.. setting:: DNS_TIMEOUT
|
||||
|
||||
DNS_TIMEOUT
|
||||
|
|
@ -734,7 +715,7 @@ Timeout for processing of DNS queries in seconds. Float is supported.
|
|||
This setting is only used by
|
||||
:class:`~scrapy.resolver.CachingThreadedResolver`. It has no effect when
|
||||
:setting:`TWISTED_REACTOR_ENABLED` is ``False``, and may have no effect
|
||||
either when :setting:`DNS_RESOLVER` is set to a different resolver.
|
||||
either when :setting:`TWISTED_DNS_RESOLVER` is set to a different resolver.
|
||||
|
||||
.. note:: This is a :ref:`reactor setting <reactor-settings>`.
|
||||
|
||||
|
|
@ -1330,6 +1311,7 @@ Default:
|
|||
|
||||
{
|
||||
"scrapy.extensions.corestats.CoreStats": 0,
|
||||
"scrapy.extensions.logcount.LogCount": 0,
|
||||
"scrapy.extensions.telnet.TelnetConsole": 0,
|
||||
"scrapy.extensions.memusage.MemoryUsage": 0,
|
||||
"scrapy.extensions.memdebug.MemoryDebugger": 0,
|
||||
|
|
@ -1352,6 +1334,8 @@ and the :ref:`list of available extensions <topics-extensions-ref>`.
|
|||
FEED_TEMPDIR
|
||||
------------
|
||||
|
||||
Default: ``None``
|
||||
|
||||
The Feed Temp dir allows you to set a custom folder to save crawler
|
||||
temporary files before uploading with :ref:`FTP feed storage <topics-feed-storage-ftp>` and
|
||||
:ref:`Amazon S3 <topics-feed-storage-s3>`.
|
||||
|
|
@ -1361,6 +1345,8 @@ temporary files before uploading with :ref:`FTP feed storage <topics-feed-storag
|
|||
FEED_STORAGE_GCS_ACL
|
||||
--------------------
|
||||
|
||||
Default: ``""``
|
||||
|
||||
The Access Control List (ACL) used when storing items to :ref:`Google Cloud Storage <topics-feed-storage-gcs>`.
|
||||
For more information on how to set this value, please refer to the column *JSON API* in `Google Cloud documentation <https://docs.cloud.google.com/storage/docs/access-control/lists>`_.
|
||||
|
||||
|
|
@ -1629,6 +1615,8 @@ The following special items are also supported:
|
|||
|
||||
- ``Python``
|
||||
|
||||
- ``pyOpenSSL``
|
||||
|
||||
.. setting:: LOGSTATS_INTERVAL
|
||||
|
||||
LOGSTATS_INTERVAL
|
||||
|
|
@ -1756,7 +1744,7 @@ significant similarities in the time between their requests.
|
|||
|
||||
The randomization policy is the same used by `wget`_ ``--random-wait`` option.
|
||||
|
||||
If :setting:`DOWNLOAD_DELAY` is zero (default) this option has no effect.
|
||||
If :setting:`DOWNLOAD_DELAY` is zero this option has no effect.
|
||||
|
||||
.. _wget: https://www.gnu.org/software/wget/manual/wget.html
|
||||
|
||||
|
|
@ -1817,7 +1805,7 @@ The parser backend to use for parsing ``robots.txt`` files. For more information
|
|||
.. setting:: ROBOTSTXT_USER_AGENT
|
||||
|
||||
ROBOTSTXT_USER_AGENT
|
||||
^^^^^^^^^^^^^^^^^^^^
|
||||
--------------------
|
||||
|
||||
Default: ``None``
|
||||
|
||||
|
|
@ -1972,6 +1960,8 @@ Default:
|
|||
|
||||
{
|
||||
"scrapy.contracts.default.UrlContract": 1,
|
||||
"scrapy.contracts.default.CallbackKeywordArgumentsContract": 1,
|
||||
"scrapy.contracts.default.MetadataContract": 1,
|
||||
"scrapy.contracts.default.ReturnsContract": 2,
|
||||
"scrapy.contracts.default.ScrapesContract": 3,
|
||||
}
|
||||
|
|
@ -2036,6 +2026,7 @@ Default:
|
|||
.. code-block:: python
|
||||
|
||||
{
|
||||
"scrapy.spidermiddlewares.start.StartSpiderMiddleware": 25,
|
||||
"scrapy.spidermiddlewares.httperror.HttpErrorMiddleware": 50,
|
||||
"scrapy.spidermiddlewares.referer.RefererMiddleware": 700,
|
||||
"scrapy.spidermiddlewares.urllength.UrlLengthMiddleware": 800,
|
||||
|
|
@ -2111,6 +2102,25 @@ command.
|
|||
The project name must not conflict with the name of custom files or directories
|
||||
in the ``project`` subdirectory.
|
||||
|
||||
.. setting:: TWISTED_DNS_RESOLVER
|
||||
|
||||
TWISTED_DNS_RESOLVER
|
||||
--------------------
|
||||
|
||||
Default: ``'scrapy.resolver.CachingThreadedResolver'``
|
||||
|
||||
The class to be used by Twisted to resolve DNS names. The default
|
||||
``scrapy.resolver.CachingThreadedResolver`` supports specifying a timeout for
|
||||
DNS requests via the :setting:`DNS_TIMEOUT` setting, but works only with IPv4
|
||||
addresses. Scrapy provides an alternative resolver,
|
||||
``scrapy.resolver.CachingHostnameResolver``, which supports IPv4/IPv6 addresses but does not
|
||||
take the :setting:`DNS_TIMEOUT` setting into account.
|
||||
|
||||
.. note::
|
||||
This setting has no effect when :setting:`TWISTED_REACTOR_ENABLED` is ``False``.
|
||||
|
||||
.. note:: This is a :ref:`reactor setting <reactor-settings>`.
|
||||
|
||||
.. setting:: TWISTED_REACTOR_ENABLED
|
||||
|
||||
TWISTED_REACTOR_ENABLED
|
||||
|
|
@ -2252,7 +2262,7 @@ URLLENGTH_LIMIT
|
|||
|
||||
Default: ``2083``
|
||||
|
||||
Scope: ``spidermiddlewares.urllength``
|
||||
Scope: ``scrapy.spidermiddlewares.urllength``
|
||||
|
||||
The maximum URL length to allow for crawled URLs.
|
||||
|
||||
|
|
|
|||
|
|
@ -60,6 +60,8 @@ Let's take an example using :ref:`coroutines <topics-coroutines>`:
|
|||
.. skip: next
|
||||
.. code-block:: python
|
||||
|
||||
import json
|
||||
|
||||
import scrapy
|
||||
import treq
|
||||
|
||||
|
|
|
|||
|
|
@ -46,7 +46,7 @@ previous (or subsequent) middleware being applied.
|
|||
If you want to disable a builtin middleware (the ones defined in
|
||||
:setting:`SPIDER_MIDDLEWARES_BASE`, and enabled by default) you must define it
|
||||
in your project :setting:`SPIDER_MIDDLEWARES` setting and assign ``None`` as its
|
||||
value. For example, if you want to disable the off-site middleware:
|
||||
value. For example, if you want to disable the referer middleware:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
|
|||
|
|
@ -208,7 +208,7 @@ scrapy.Spider
|
|||
:param response: the response to parse
|
||||
:type response: :class:`~scrapy.http.Response`
|
||||
|
||||
.. method:: log(message, [level, component])
|
||||
.. method:: log(message, [level])
|
||||
|
||||
Wrapper that sends a log message through the Spider's :attr:`logger`,
|
||||
kept for backward compatibility. For more information see
|
||||
|
|
@ -335,8 +335,8 @@ The above example can also be written as follows:
|
|||
|
||||
If you are :ref:`running Scrapy from a script <run-from-script>`, you can
|
||||
specify spider arguments when calling
|
||||
:class:`CrawlerProcess.crawl <scrapy.crawler.CrawlerProcess.crawl>` or
|
||||
:class:`CrawlerRunner.crawl <scrapy.crawler.CrawlerRunner.crawl>`:
|
||||
:meth:`CrawlerProcess.crawl <scrapy.crawler.CrawlerProcess.crawl>` or
|
||||
:meth:`CrawlerRunner.crawl <scrapy.crawler.CrawlerRunner.crawl>`:
|
||||
|
||||
.. skip: next
|
||||
.. code-block:: python
|
||||
|
|
@ -593,7 +593,7 @@ Let's now take a look at an example CrawlSpider with rules:
|
|||
This spider would start crawling example.com's home page, collecting category
|
||||
links, and item links, parsing the latter with the ``parse_item`` method. For
|
||||
each item response, some data will be extracted from the HTML using XPath, and
|
||||
an :class:`~scrapy.Item` will be filled with it.
|
||||
a dictionary will be filled with it.
|
||||
|
||||
XMLFeedSpider
|
||||
-------------
|
||||
|
|
@ -953,6 +953,7 @@ Combine SitemapSpider with other sources of urls:
|
|||
|
||||
.. code-block:: python
|
||||
|
||||
from scrapy import Request
|
||||
from scrapy.spiders import SitemapSpider
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -36,9 +36,9 @@ disable it if you want. For more information about the extension itself see
|
|||
How to access the telnet console
|
||||
================================
|
||||
|
||||
The telnet console listens in the TCP port defined in the
|
||||
:setting:`TELNETCONSOLE_PORT` setting, which defaults to ``6023``. To access
|
||||
the console you need to type::
|
||||
The telnet console listens on the first available TCP port from the range
|
||||
defined in the :setting:`TELNETCONSOLE_PORT` setting, which defaults to
|
||||
``[6023, 6073]``. To access the console you need to type::
|
||||
|
||||
telnet localhost 6023
|
||||
Trying localhost...
|
||||
|
|
|
|||
|
|
@ -39,8 +39,8 @@ API stability
|
|||
|
||||
API stability was one of the major goals for the *1.0* release.
|
||||
|
||||
Methods or functions that start with a single dash (``_``) are private and
|
||||
should never be relied as stable.
|
||||
Methods or functions that start with a single underscore (``_``) are private
|
||||
and should never be relied upon as stable.
|
||||
|
||||
Also, keep in mind that stable doesn't mean complete: stable APIs could grow
|
||||
new methods or functionality but the existing methods should keep working the
|
||||
|
|
|
|||
|
|
@ -240,7 +240,7 @@ BOT_NAME = "scrapybot"
|
|||
CLOSESPIDER_ERRORCOUNT = 0
|
||||
CLOSESPIDER_ITEMCOUNT = 0
|
||||
CLOSESPIDER_PAGECOUNT = 0
|
||||
CLOSESPIDER_TIMEOUT = 0
|
||||
CLOSESPIDER_TIMEOUT = 0.0
|
||||
CLOSESPIDER_PAGECOUNT_NO_ITEM = 0
|
||||
CLOSESPIDER_TIMEOUT_NO_ITEM = 0
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue