RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #4090 from Gallaecio/documentation-build 

Fix references to Python types in parameter type fields
This commit is contained in:
Mikhail Korobov 2020-08-14 00:58:06 +05:00 committed by GitHub
parent cd0d3fd48d b2f4df5cb7
commit 61653418ff
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
10 changed files with 67 additions and 67 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
docs/topics/contracts.rst
View File

@ -81,7 +81,7 @@ override three methods:
.. class:: Contract(method, *args)
:param method: callback function to which the contract is associated
:type method: function
:type method: collections.abc.Callable
:param args: list of arguments passed into the docstring (whitespace
separated)

14
docs/topics/email.rst
View File

@ -62,10 +62,10 @@ rest of the framework.
:type smtpport: int
:param smtptls: enforce using SMTP STARTTLS
:type smtptls: boolean
:type smtptls: bool
:param smtpssl: enforce using a secure SSL connection
:type smtpssl: boolean
:type smtpssl: bool
.. classmethod:: from_settings(settings)
@ -79,14 +79,14 @@ rest of the framework.
Send email to the given recipients.
:param to: the e-mail recipients
:type to: str or list of str
:param to: the e-mail recipients as a string or as a list of strings
:type to: str or list
:param subject: the subject of the e-mail
:type subject: str
:param cc: the e-mails to CC
:type cc: str or list of str
:param cc: the e-mails to CC as a string or as a list of strings
:type cc: str or list
:param body: the e-mail body
:type body: str
@ -96,7 +96,7 @@ rest of the framework.
appear on the e-mail's attachment, ``mimetype`` is the mimetype of the
attachment and ``file_object`` is a readable file object with the
contents of the attachment
:type attachs: iterable
:type attachs: collections.abc.Iterable
:param mimetype: the MIME type of the e-mail
:type mimetype: str

2
docs/topics/exporters.rst
View File

@ -305,7 +305,7 @@ CsvItemExporter
:param include_headers_line: If enabled, makes the exporter output a header
line with the field names taken from
:attr:`BaseItemExporter.fields_to_export` or the first exported item fields.
:type include_headers_line: boolean
:type include_headers_line: bool
:param join_multivalued: The char (or chars) that will be used for joining
multi-valued fields, if found.

2
docs/topics/leaks.rst
View File

@ -179,7 +179,7 @@ Here are the functions available in the :mod:`~scrapy.utils.trackref` module.
:param ignore: if given, all objects from the specified class (or tuple of
classes) will be ignored.
:type ignore: class or classes tuple
:type ignore: type or tuple
.. function:: get_oldest(class_name)

14
docs/topics/link-extractors.rst
View File

@ -46,13 +46,13 @@ LxmlLinkExtractor
:param allow: a single regular expression (or list of regular expressions)
that the (absolute) urls must match in order to be extracted. If not
given (or empty), it will match all links.
:type allow: a regular expression (or list of)
:type allow: str or list
:param deny: a single regular expression (or list of regular expressions)
that the (absolute) urls must match in order to be excluded (i.e. not
extracted). It has precedence over the ``allow`` parameter. If not
given (or empty) it won't exclude any links.
:type deny: a regular expression (or list of)
:type deny: str or list
:param allow_domains: a single value or a list of string containing
domains which will be considered for extracting the links
@ -88,7 +88,7 @@ LxmlLinkExtractor
that the link's text must match in order to be extracted. If not
given (or empty), it will match all links. If a list of regular expressions is
given, the link will be extracted if it matches at least one.
:type restrict_text: a regular expression (or list of)
:type restrict_text: str or list
:param tags: a tag or a list of tags to consider when extracting links.
Defaults to ``('a', 'area')``.
@ -106,11 +106,11 @@ LxmlLinkExtractor
different for requests with canonicalized and raw URLs. If you're
using LinkExtractor to follow links it is more robust to
keep the default ``canonicalize=False``.
:type canonicalize: boolean
:type canonicalize: bool
:param unique: whether duplicate filtering should be applied to extracted
links.
:type unique: boolean
:type unique: bool
:param process_value: a function which receives each value extracted from
the tag and attributes scanned and can modify the value and return a
@ -132,7 +132,7 @@ LxmlLinkExtractor
if m:
return m.group(1)
:type process_value: callable
:type process_value: collections.abc.Callable
:param strip: whether to strip whitespaces from extracted attributes.
According to HTML5 standard, leading and trailing whitespaces
@ -141,7 +141,7 @@ LxmlLinkExtractor
elements, etc., so LinkExtractor strips space chars by default.
Set ``strip=False`` to turn it off (e.g. if you're extracting urls
from elements or attributes which allow leading/trailing whitespaces).
:type strip: boolean
:type strip: bool
.. automethod:: extract_links

52
docs/topics/request-response.rst
View File

@ -33,7 +33,7 @@ Request objects
:param url: the URL of this request
If the URL is invalid, a :exc:`ValueError` exception is raised.
:type url: string
:type url: str
:param callback: the function that will be called with the response of this
request (once it's downloaded) as its first parameter. For more information
@ -42,10 +42,10 @@ Request objects
:meth:`~scrapy.spiders.Spider.parse` method will be used.
Note that if exceptions are raised during processing, errback is called instead.
:type callback: callable
:type callback: collections.abc.Callable
:param method: the HTTP method of this request. Defaults to ``'GET'``.
:type method: string
:type method: str
:param meta: the initial values for the :attr:`Request.meta` attribute. If
given, the dict passed in this parameter will be shallow copied.
@ -107,7 +107,7 @@ Request objects
:param encoding: the encoding of this request (defaults to ``'utf-8'``).
This encoding will be used to percent-encode the URL and to convert the
body to bytes (if given as a string).
:type encoding: string
:type encoding: str
:param priority: the priority of this request (defaults to ``0``).
The priority is used by the scheduler to define the order used to process
@ -119,7 +119,7 @@ Request objects
the scheduler. This is used when you want to perform an identical
request multiple times, to ignore the duplicates filter. Use it with
care, or you will get into crawling loops. Default to ``False``.
:type dont_filter: boolean
:type dont_filter: bool
:param errback: a function that will be called if any exception was
raised while processing the request. This includes pages that failed
@ -131,7 +131,7 @@ Request objects
.. versionchanged:: 2.0
The *callback* parameter is no longer required when the *errback*
parameter is specified.
:type errback: callable
:type errback: collections.abc.Callable
:param flags: Flags sent to the request, can be used for logging or similar purposes.
:type flags: list
@ -159,7 +159,7 @@ Request objects
.. attribute:: Request.body
A str that contains the request body.
The request body as bytes.
This attribute is read-only. To change the body of a Request use
:meth:`replace`.
@ -485,7 +485,7 @@ fields with form data from :class:`Response` objects.
:param formdata: is a dictionary (or iterable of (key, value) tuples)
containing HTML Form data which will be url-encoded and assigned to the
body of the request.
:type formdata: dict or iterable of tuples
:type formdata: dict or collections.abc.Iterable
The :class:`FormRequest` objects support the following class method in
addition to the standard :class:`Request` methods:
@ -517,20 +517,20 @@ fields with form data from :class:`Response` objects.
:type response: :class:`Response` object
:param formname: if given, the form with name attribute set to this value will be used.
:type formname: string
:type formname: str
:param formid: if given, the form with id attribute set to this value will be used.
:type formid: string
:type formid: str
:param formxpath: if given, the first form that matches the xpath will be used.
:type formxpath: string
:type formxpath: str
:param formcss: if given, the first form that matches the css selector will be used.
:type formcss: string
:type formcss: str
:param formnumber: the number of form to use, when the response contains
multiple forms. The first one (and also the default) is ``0``.
:type formnumber: integer
:type formnumber: int
:param formdata: fields to override in the form data. If a field was
already present in the response ``<form>`` element, its value is
@ -548,7 +548,7 @@ fields with form data from :class:`Response` objects.
:param dont_click: If True, the form data will be submitted without
clicking in any element.
:type dont_click: boolean
:type dont_click: bool
The other parameters of this class method are passed directly to the
:class:`FormRequest` ``__init__`` method.
@ -636,7 +636,7 @@ dealing with JSON requests.
if :attr:`Request.body` argument is provided this parameter will be ignored.
if :attr:`Request.body` argument is not provided and data argument is provided :attr:`Request.method` will be
set to ``'POST'`` automatically.
:type data: JSON serializable object
:type data: object
:param dumps_kwargs: Parameters that will be passed to underlying :func:`json.dumps` method which is used to serialize
data into JSON format.
@ -663,16 +663,16 @@ Response objects
downloaded (by the Downloader) and fed to the Spiders for processing.
:param url: the URL of this response
:type url: string
:type url: str
:param status: the HTTP status of the response. Defaults to ``200``.
:type status: integer
:type status: int
:param headers: the headers of this response. The dict values can be strings
(for single valued headers) or lists (for multi-valued headers).
:type headers: dict
:param body: the response body. To access the decoded text as str you can use
:param body: the response body. To access the decoded text as a string, use
``response.text`` from an encoding-aware
:ref:`Response subclass <topics-request-response-ref-response-subclasses>`,
such as :class:`TextResponse`.
@ -720,10 +720,10 @@ Response objects
.. attribute:: Response.body
The body of this Response. Keep in mind that Response.body
is always a bytes object. If you want the string version use
:attr:`TextResponse.text` (only available in :class:`TextResponse`
and subclasses).
The response body as bytes.
If you want the body as a string, use :attr:`TextResponse.text` (only
available in :class:`TextResponse` and subclasses).
This attribute is read-only. To change the body of a Response use
:meth:`replace`.
@ -843,10 +843,10 @@ TextResponse objects
:param encoding: is a string which contains the encoding to use for this
response. If you create a :class:`TextResponse` object with a string as
body, it will be encoded using this encoding (remember the body attribute
is always a bytes object). If ``encoding`` is ``None`` (default value), the
encoding will be looked up in the response headers and body instead.
:type encoding: string
body, it will be converted to bytes encoded using this encoding. If
*encoding* is ``None`` (default), the encoding will be looked up in the
response headers and body instead.
:type encoding: str
:class:`TextResponse` objects support the following attributes in addition
to the standard :class:`Response` ones:

2
scrapy/crawler.py
View File

@ -307,7 +307,7 @@ class CrawlerProcess(CrawlerRunner):
If ``stop_after_crawl`` is True, the reactor will be stopped after all
crawlers have finished, using :meth:`join`.
:param boolean stop_after_crawl: stop or not the reactor when all
:param bool stop_after_crawl: stop or not the reactor when all
crawlers have finished
"""
from twisted.internet import reactor

4
scrapy/robotstxt.py
View File

@ -47,10 +47,10 @@ class RobotParser(metaclass=ABCMeta):
"""Return ``True`` if ``user_agent`` is allowed to crawl ``url``, otherwise return ``False``.
:param url: Absolute URL
:type url: string
:type url: str
:param user_agent: User agent
:type user_agent: string
:type user_agent: str
"""
pass

40
scrapy/settings/__init__.py
View File

@ -98,10 +98,10 @@ class BaseSettings(MutableMapping):
Get a setting value without affecting its original type.
:param name: the setting name
:type name: string
:type name: str
:param default: the value to return if no setting is found
:type default: any
:type default: object
"""
return self[name] if self[name] is not None else default
@ -116,10 +116,10 @@ class BaseSettings(MutableMapping):
``'0'`` will return ``False`` when using this method.
:param name: the setting name
:type name: string
:type name: str
:param default: the value to return if no setting is found
:type default: any
:type default: object
"""
got = self.get(name, default)
try:
@ -138,10 +138,10 @@ class BaseSettings(MutableMapping):
Get a setting value as an int.
:param name: the setting name
:type name: string
:type name: str
:param default: the value to return if no setting is found
:type default: any
:type default: object
"""
return int(self.get(name, default))
@ -150,10 +150,10 @@ class BaseSettings(MutableMapping):
Get a setting value as a float.
:param name: the setting name
:type name: string
:type name: str
:param default: the value to return if no setting is found
:type default: any
:type default: object
"""
return float(self.get(name, default))
@ -166,10 +166,10 @@ class BaseSettings(MutableMapping):
``'one,two'`` will return a list ['one', 'two'] when using this method.
:param name: the setting name
:type name: string
:type name: str
:param default: the value to return if no setting is found
:type default: any
:type default: object
"""
value = self.get(name, default or [])
if isinstance(value, str):
@ -187,10 +187,10 @@ class BaseSettings(MutableMapping):
and losing all information about priority and mutability.
:param name: the setting name
:type name: string
:type name: str
:param default: the value to return if no setting is found
:type default: any
:type default: object
"""
value = self.get(name, default or {})
if isinstance(value, str):
@ -202,7 +202,7 @@ class BaseSettings(MutableMapping):
counterpart.
:param name: name of the dictionary-like setting
:type name: string
:type name: str
"""
compbs = BaseSettings()
compbs.update(self[name + '_BASE'])
@ -215,7 +215,7 @@ class BaseSettings(MutableMapping):
the given ``name`` does not exist.
:param name: the setting name
:type name: string
:type name: str
"""
if name not in self:
return None
@ -245,14 +245,14 @@ class BaseSettings(MutableMapping):
otherwise they won't have any effect.
:param name: the setting name
:type name: string
:type name: str
:param value: the value to associate with the setting
:type value: any
:type value: object
:param priority: the priority of the setting. Should be a key of
:attr:`~scrapy.settings.SETTINGS_PRIORITIES` or an integer
:type priority: string or int
:type priority: str or int
"""
self._assert_mutability()
priority = get_settings_priority(priority)
@ -276,11 +276,11 @@ class BaseSettings(MutableMapping):
uppercase variable of ``module`` with the provided ``priority``.
:param module: the module or the path of the module
:type module: module object or string
:type module: types.ModuleType or str
:param priority: the priority of the settings. Should be a key of
:attr:`~scrapy.settings.SETTINGS_PRIORITIES` or an integer
:type priority: string or int
:type priority: str or int
"""
self._assert_mutability()
if isinstance(module, str):
@ -309,7 +309,7 @@ class BaseSettings(MutableMapping):
:param priority: the priority of the settings. Should be a key of
:attr:`~scrapy.settings.SETTINGS_PRIORITIES` or an integer
:type priority: string or int
:type priority: str or int
"""
self._assert_mutability()
if isinstance(values, str):

2
scrapy/signalmanager.py
View File

@ -16,7 +16,7 @@ class SignalManager:
section.
:param receiver: the function to be connected
:type receiver: callable
:type receiver: collections.abc.Callable
:param signal: the signal to connect to
:type signal: object