mirror of https://github.com/scrapy/scrapy.git
[docs] refactor python docs links using intersphinx
This commit is contained in:
parent
f37b1bdc56
commit
80c69d68ad
|
|
@ -287,8 +287,8 @@ to be scraped, you can at least get **some** data.
|
|||
|
||||
Besides the :meth:`~scrapy.selector.SelectorList.getall` and
|
||||
:meth:`~scrapy.selector.SelectorList.get` methods, you can also use
|
||||
the :meth:`~scrapy.selector.SelectorList.re` method to extract using `regular
|
||||
expressions`_:
|
||||
the :meth:`~scrapy.selector.SelectorList.re` method to extract using
|
||||
:doc:`regular expressions <library/re>`:
|
||||
|
||||
>>> response.css('title::text').re(r'Quotes.*')
|
||||
['Quotes to Scrape']
|
||||
|
|
@ -305,7 +305,6 @@ with a selector (see :ref:`topics-developer-tools`).
|
|||
`Selector Gadget`_ is also a nice tool to quickly find CSS selector for
|
||||
visually selected elements, which works in many browsers.
|
||||
|
||||
.. _regular expressions: https://docs.python.org/3/library/re.html
|
||||
.. _Selector Gadget: https://selectorgadget.com/
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -76,8 +76,8 @@ becomes::
|
|||
|
||||
Coroutines may be used to call asynchronous code. This includes other
|
||||
coroutines, functions that return Deferreds and functions that return
|
||||
`awaitable objects`_ such as :class:`~asyncio.Future`. This means you can use
|
||||
many useful Python libraries providing such code::
|
||||
:term:`awaitable objects <awaitable>` such as :class:`~asyncio.Future`.
|
||||
This means you can use many useful Python libraries providing such code::
|
||||
|
||||
class MySpider(Spider):
|
||||
# ...
|
||||
|
|
@ -107,4 +107,3 @@ Common use cases for asynchronous code include:
|
|||
:ref:`the screenshot pipeline example<ScreenshotPipeline>`).
|
||||
|
||||
.. _aio-libs: https://github.com/aio-libs
|
||||
.. _awaitable objects: https://docs.python.org/3/glossary.html#term-awaitable
|
||||
|
|
|
|||
|
|
@ -980,7 +980,7 @@ RobotsTxtMiddleware
|
|||
Scrapy ships with support for the following robots.txt_ parsers:
|
||||
|
||||
* :ref:`Protego <protego-parser>` (default)
|
||||
* :class:`~urllib.robotparser.RobotFileParser`
|
||||
* :ref:`RobotFileParser <python-robotfileparser>`
|
||||
* :ref:`Reppy <reppy-parser>`
|
||||
* :ref:`Robotexclusionrulesparser <rerp-parser>`
|
||||
|
||||
|
|
@ -1028,6 +1028,8 @@ Based on `Protego <https://github.com/scrapy/protego>`_:
|
|||
|
||||
Scrapy uses this parser by default.
|
||||
|
||||
.. _python-robotfileparser:
|
||||
|
||||
RobotFileParser
|
||||
~~~~~~~~~~~~~~~
|
||||
|
||||
|
|
|
|||
|
|
@ -130,8 +130,9 @@ data from it depends on the type of response:
|
|||
- If the response is JavaScript, or HTML with a ``<script/>`` element
|
||||
containing the desired data, see :ref:`topics-parsing-javascript`.
|
||||
|
||||
- If the response is CSS, use :mod:`re` to extract the desired
|
||||
data from :attr:`response.text <scrapy.http.TextResponse.text>`.
|
||||
- If the response is CSS, use a :doc:`regular expression <library/re>` to
|
||||
extract the desired data from
|
||||
:attr:`response.text <scrapy.http.TextResponse.text>`.
|
||||
|
||||
.. _topics-parsing-images:
|
||||
|
||||
|
|
@ -168,8 +169,9 @@ JavaScript code:
|
|||
Once you have a string with the JavaScript code, you can extract the desired
|
||||
data from it:
|
||||
|
||||
- You might be able to use :mod:`re` to extract the desired
|
||||
data in JSON format, which you can then parse with :func:`json.loads`.
|
||||
- You might be able to use a :doc:`regular expression <library/re>` to
|
||||
extract the desired data in JSON format, which you can then parse with
|
||||
:func:`json.loads`.
|
||||
|
||||
For example, if the JavaScript code contains a separate line like
|
||||
``var data = {"field": "value"};`` you can extract that data as follows:
|
||||
|
|
|
|||
|
|
@ -311,7 +311,7 @@ CsvItemExporter
|
|||
|
||||
The additional keyword arguments of this ``__init__`` method are passed to the
|
||||
:class:`BaseItemExporter` ``__init__`` method, and the leftover arguments to the
|
||||
:func:`csv.writer` ``__init__`` method, so you can use any ``csv.writer`` ``__init__`` method
|
||||
:func:`csv.writer` function, so you can use any :func:`csv.writer` function
|
||||
argument to customize this exporter.
|
||||
|
||||
A typical output of this exporter would be::
|
||||
|
|
@ -333,7 +333,7 @@ PickleItemExporter
|
|||
:param protocol: The pickle protocol to use.
|
||||
:type protocol: int
|
||||
|
||||
For more information, refer :mod:`pickle`.
|
||||
For more information, see :mod:`pickle`.
|
||||
|
||||
The additional keyword arguments of this ``__init__`` method are passed to the
|
||||
:class:`BaseItemExporter` ``__init__`` method.
|
||||
|
|
|
|||
|
|
@ -364,7 +364,7 @@ Debugger extension
|
|||
|
||||
.. class:: Debugger
|
||||
|
||||
Invokes a :mod:`Python debugger <pdb>`: inside a running Scrapy process when a `SIGUSR2`_
|
||||
Invokes a :doc:`Python debugger <library/pdb>` inside a running Scrapy process when a `SIGUSR2`_
|
||||
signal is received. After the debugger is exited, the Scrapy process continues
|
||||
running normally.
|
||||
|
||||
|
|
|
|||
|
|
@ -15,8 +15,8 @@ especially in a larger project with many spiders.
|
|||
|
||||
To define common output data format Scrapy provides the :class:`Item` class.
|
||||
:class:`Item` objects are simple containers used to collect the scraped data.
|
||||
They provide a :class:`dict` like API with a convenient syntax for declaring
|
||||
their available fields.
|
||||
They provide an API similar to :class:`dict` API with a convenient syntax
|
||||
for declaring their available fields.
|
||||
|
||||
Various Scrapy components use extra information provided by Items:
|
||||
exporters look at declared fields to figure out columns to export,
|
||||
|
|
@ -143,7 +143,7 @@ KeyError: 'Product does not support field: lala'
|
|||
Accessing all populated values
|
||||
------------------------------
|
||||
|
||||
To access all populated values, just use the typical :class:`dict`:
|
||||
To access all populated values, just use the typical :class:`dict` API:
|
||||
|
||||
>>> product.keys()
|
||||
['price', 'name']
|
||||
|
|
@ -160,11 +160,9 @@ Copying items
|
|||
To copy an item, you must first decide whether you want a shallow copy or a
|
||||
deep copy.
|
||||
|
||||
If your item contains mutable_ values like lists or dictionaries, a shallow
|
||||
copy will keep references to the same mutable values across all different
|
||||
copies.
|
||||
|
||||
.. _mutable: https://docs.python.org/3/glossary.html#term-mutable
|
||||
If your item contains :term:`mutable` values like lists or dictionaries,
|
||||
a shallow copy will keep references to the same mutable values across all
|
||||
different copies.
|
||||
|
||||
For example, if you have an item with a list of tags, and you create a shallow
|
||||
copy of that item, both the original item and the copy have the same list of
|
||||
|
|
@ -231,8 +229,8 @@ Item objects
|
|||
|
||||
Return a new Item optionally initialized from the given argument.
|
||||
|
||||
Items replicate the standard :class:`dict`, including its ``__init__`` method, and
|
||||
also provide the following additional API members:
|
||||
Items replicate the standard :class:`dict` API, including its ``__init__``
|
||||
method, and also provide the following additional API members:
|
||||
|
||||
.. automethod:: copy
|
||||
|
||||
|
|
|
|||
|
|
@ -82,10 +82,10 @@ path::
|
|||
|
||||
.. seealso::
|
||||
|
||||
Module logging, `HowTo <https://docs.python.org/3/howto/logging.html>`_
|
||||
Module logging, :doc:`HowTo <howto/logging>`
|
||||
Basic Logging Tutorial
|
||||
|
||||
Module logging, :class:`~logging.Logger`
|
||||
Module logging, :ref:`Loggers <logger>`
|
||||
Further documentation on loggers
|
||||
|
||||
.. _topics-logging-from-spiders:
|
||||
|
|
@ -164,10 +164,8 @@ possible levels listed in :ref:`topics-logging-levels`.
|
|||
|
||||
:setting:`LOG_FORMAT` and :setting:`LOG_DATEFORMAT` specify formatting strings
|
||||
used as layouts for all messages. Those strings can contain any placeholders
|
||||
listed in `logging's logrecord attributes docs
|
||||
<https://docs.python.org/3/library/logging.html#logrecord-attributes>`_ and
|
||||
`datetime's strftime and strptime directives
|
||||
<https://docs.python.org/3/library/datetime.html#strftime-and-strptime-behavior>`_
|
||||
listed in :ref:`logging's logrecord attributes docs <logrecord-attributes>` and
|
||||
:ref:`datetime's strftime and strptime directives <strftime-strptime-behavior>`
|
||||
respectively.
|
||||
|
||||
If :setting:`LOG_SHORT_NAMES` is set, then the logs will not display the Scrapy
|
||||
|
|
|
|||
|
|
@ -174,9 +174,9 @@ Request objects
|
|||
See :ref:`topics-request-meta` for a list of special meta keys
|
||||
recognized by Scrapy.
|
||||
|
||||
This dict is `shallow copied`_ when the request is cloned using the
|
||||
``copy()`` or ``replace()`` methods, and can also be accessed, in your
|
||||
spider, from the ``response.meta`` attribute.
|
||||
This dict is :mod:`shallow copied <copy>` when the request is
|
||||
cloned using the ``copy()`` or ``replace()`` methods, and can also be
|
||||
accessed, in your spider, from the ``response.meta`` attribute.
|
||||
|
||||
.. attribute:: Request.cb_kwargs
|
||||
|
||||
|
|
@ -185,11 +185,9 @@ Request objects
|
|||
for new Requests, which means by default callbacks only get a :class:`Response`
|
||||
object as argument.
|
||||
|
||||
This dict is `shallow copied`_ when the request is cloned using the
|
||||
``copy()`` or ``replace()`` methods, and can also be accessed, in your
|
||||
spider, from the ``response.cb_kwargs`` attribute.
|
||||
|
||||
.. _shallow copied: https://docs.python.org/3/library/copy.html
|
||||
This dict is :mod:`shallow copied <copy>` when the request is
|
||||
cloned using the ``copy()`` or ``replace()`` methods, and can also be
|
||||
accessed, in your spider, from the ``response.cb_kwargs`` attribute.
|
||||
|
||||
.. method:: Request.copy()
|
||||
|
||||
|
|
|
|||
|
|
@ -26,9 +26,7 @@ do this by using an environment variable, ``SCRAPY_SETTINGS_MODULE``.
|
|||
|
||||
The value of ``SCRAPY_SETTINGS_MODULE`` should be in Python path syntax, e.g.
|
||||
``myproject.settings``. Note that the settings module should be on the
|
||||
Python `import search path`_.
|
||||
|
||||
.. _import search path: https://docs.python.org/3/tutorial/modules.html#the-module-search-path
|
||||
Python :ref:`import search path <tut-searchpath>`.
|
||||
|
||||
.. _populating-settings:
|
||||
|
||||
|
|
@ -899,10 +897,9 @@ LOG_FORMAT
|
|||
|
||||
Default: ``'%(asctime)s [%(name)s] %(levelname)s: %(message)s'``
|
||||
|
||||
String for formatting log messages. Refer to the `Python logging documentation`_ for the whole list of available
|
||||
placeholders.
|
||||
|
||||
.. _Python logging documentation: https://docs.python.org/3/library/logging.html#logrecord-attributes
|
||||
String for formatting log messages. Refer to the
|
||||
:ref:`Python logging documentation <logrecord-attributes>` for the qwhole
|
||||
list of available placeholders.
|
||||
|
||||
.. setting:: LOG_DATEFORMAT
|
||||
|
||||
|
|
@ -912,10 +909,9 @@ LOG_DATEFORMAT
|
|||
Default: ``'%Y-%m-%d %H:%M:%S'``
|
||||
|
||||
String for formatting date/time, expansion of the ``%(asctime)s`` placeholder
|
||||
in :setting:`LOG_FORMAT`. Refer to the `Python datetime documentation`_ for the whole list of available
|
||||
directives.
|
||||
|
||||
.. _Python datetime documentation: https://docs.python.org/3/library/datetime.html#strftime-and-strptime-behavior
|
||||
in :setting:`LOG_FORMAT`. Refer to the
|
||||
:ref:`Python datetime documentation <strftime-strptime-behavior>` for the
|
||||
whole list of available directives.
|
||||
|
||||
.. setting:: LOG_FORMATTER
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue