diff --git a/docs/contributing.rst b/docs/contributing.rst index c868a0ac4..68cd1672c 100644 --- a/docs/contributing.rst +++ b/docs/contributing.rst @@ -308,6 +308,17 @@ In any case, if something is covered in a docstring, use the documentation instead of duplicating the docstring in files within the ``docs/`` directory. +When adding a new setting to ``scrapy/settings/default_settings.py``: + +* add it in alphabetical order, with the exception that enabling flags and + other high-level settings for a group should come first in their group + and pairs like host/port and user/password should be in the usual order +* group similar settings without leaving blank lines +* add its documentation to :doc:`topics/settings` + +When adding a new exception or signal, document it in :doc:`topics/exceptions` +or :doc:`topics/signals`, respectively. + Documentation updates that cover new or modified features must use Sphinx’s :rst:dir:`versionadded` and :rst:dir:`versionchanged` directives. Use ``VERSION`` as version, we will replace it with the actual version right before diff --git a/docs/topics/download-handlers.rst b/docs/topics/download-handlers.rst index c901e01e4..310b494fe 100644 --- a/docs/topics/download-handlers.rst +++ b/docs/topics/download-handlers.rst @@ -378,6 +378,25 @@ This handler supports ``ftp://host/path`` FTP URIs. It's implemented using :mod:`twisted.protocols.ftp`. +FTP connection parameters can be passed using :attr:`Request.meta +`: + +* ``ftp_user`` (falls back to :setting:`FTP_USER`) +* ``ftp_password`` (falls back to :setting:`FTP_PASSWORD`) +* ``ftp_passive`` (falls back to :setting:`FTP_PASSIVE_MODE`) +* ``ftp_local_filename``: if not given, file data is returned in + ``response.body``. If given, file data is saved to that local path (useful + for large files) and the local filename is also returned in ``response.body``. + +The built response uses HTTP-like status codes by default: ``200`` on success, +``404`` when the file is not found (FTP code 550), or the corresponding FTP +exception otherwise. The mapping from FTP return codes to response status codes +is defined in the :attr:`~scrapy.core.downloader.handlers.ftp.FTPDownloadHandler.CODE_MAPPING` +class attribute. The ``default`` key is used for unmapped codes. + +On successful requests (status 200), ``response.headers`` includes +``Local Filename`` (when ``ftp_local_filename`` is used) and ``Size``. + S3DownloadHandler ----------------- diff --git a/docs/topics/exceptions.rst b/docs/topics/exceptions.rst index cc690d609..4508d6e60 100644 --- a/docs/topics/exceptions.rst +++ b/docs/topics/exceptions.rst @@ -7,6 +7,11 @@ Exceptions .. module:: scrapy.exceptions :synopsis: Scrapy exceptions +.. note:: + + When adding a new exception to ``scrapy.exceptions``, document it on this + page. + .. _topics-exceptions-ref: Built-in Exceptions reference diff --git a/docs/topics/leaks.rst b/docs/topics/leaks.rst index 0913a3310..32286ab23 100644 --- a/docs/topics/leaks.rst +++ b/docs/topics/leaks.rst @@ -182,6 +182,13 @@ scrapy.utils.trackref module Here are the functions available in the :mod:`~scrapy.utils.trackref` module. +.. note:: + + PyPy uses a tracing garbage collector, so objects may remain in + ``live_refs`` longer than expected, even after they go out of scope. If + deterministic behavior is required, you may need to explicitly trigger + garbage collection or call ``trackref.live_refs.clear()``. + .. class:: object_ref Inherit from this class if you want to track live diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rst index f2fcfb0b5..7c84df2f0 100644 --- a/docs/topics/request-response.rst +++ b/docs/topics/request-response.rst @@ -720,6 +720,8 @@ Those are: * :reqmeta:`download_warnsize` * :reqmeta:`download_timeout` * ``ftp_password`` (See :setting:`FTP_PASSWORD` for more info) +* ``ftp_passive`` (See :setting:`FTP_PASSIVE_MODE` for more info) +* ``ftp_local_filename`` (See :ref:`topics-download-handlers` for more info) * ``ftp_user`` (See :setting:`FTP_USER` for more info) * :reqmeta:`give_up_log_level` * :reqmeta:`handle_httpstatus_all` diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst index 11251a92c..005bedcc9 100644 --- a/docs/topics/settings.rst +++ b/docs/topics/settings.rst @@ -2056,6 +2056,18 @@ Example: .. note:: This is a :ref:`pre-crawler setting `. +.. setting:: STATSMAILER_RCPTS + +STATSMAILER_RCPTS +----------------- + +Default: ``[]`` + +A list of email addresses to receive a message with scraping stats when a +spider finishes. Used by the deprecated +:class:`~scrapy.extensions.statsmailer.StatsMailer` extension, which is enabled +when this setting is non-empty. + .. setting:: STATS_CLASS STATS_CLASS diff --git a/docs/topics/signals.rst b/docs/topics/signals.rst index d13733623..d0ed7110a 100644 --- a/docs/topics/signals.rst +++ b/docs/topics/signals.rst @@ -16,6 +16,10 @@ deliver the arguments that the handler receives. You can connect to signals (or send your own) through the :ref:`topics-api-signals`. +.. note:: + + When adding a new signal to ``scrapy.signals``, document it on this page. + Here is a simple example showing how you can catch signals and perform some action: .. code-block:: python diff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rst index 506daf930..11bce19cd 100644 --- a/docs/topics/spiders.rst +++ b/docs/topics/spiders.rst @@ -787,6 +787,11 @@ SitemapSpider SitemapSpider allows you to crawl a site by discovering the URLs using `Sitemaps`_. + .. note:: + + The :mod:`scrapy.utils.sitemap` module provides sitemap parsing support + for this spider. Its API is subject to change without notice. + It supports nested sitemaps and discovering sitemap urls from `robots.txt`_. diff --git a/scrapy/core/downloader/handlers/_httpx.py b/scrapy/core/downloader/handlers/_httpx.py index 9b54b44d8..c7aebf5c8 100644 --- a/scrapy/core/downloader/handlers/_httpx.py +++ b/scrapy/core/downloader/handlers/_httpx.py @@ -1,5 +1,3 @@ -"""``httpx``-based HTTP(S) download handler. Currently not recommended for production use.""" - from __future__ import annotations import ipaddress diff --git a/scrapy/core/downloader/handlers/ftp.py b/scrapy/core/downloader/handlers/ftp.py index 07ff4a74e..633b263e8 100644 --- a/scrapy/core/downloader/handlers/ftp.py +++ b/scrapy/core/downloader/handlers/ftp.py @@ -1,33 +1,3 @@ -""" -An asynchronous FTP file download handler for scrapy which somehow emulates an http response. - -FTP connection parameters are passed using the request meta field: -- ftp_user (optional, falls back to FTP_USER) -- ftp_password (optional, falls back to FTP_PASSWORD) -- ftp_passive (optional, falls back to FTP_PASSIVE_MODE) sets FTP connection passive mode -- ftp_local_filename - - If not given, file data will come in the response.body, as a normal scrapy Response, - which will imply that the entire file will be on memory. - - if given, file data will be saved in a local file with the given name - This helps when downloading very big files to avoid memory issues. In addition, for - convenience the local file name will also be given in the response body. - -The status of the built html response will be, by default -- 200 in case of success -- 404 in case specified file was not found in the server (ftp code 550) - -or raise corresponding ftp exception otherwise - -The matching from server ftp command return codes to html response codes is defined in the -CODE_MAPPING attribute of the handler class. The key 'default' is used for any code -that is not explicitly present among the map keys. You may need to overwrite this -mapping if want a different behaviour than default. - -In case of status 200 request, response.headers will come with two keys: - 'Local Filename' - with the value of the local filename if given - 'Size' - with size of the downloaded data -""" - from __future__ import annotations import re diff --git a/scrapy/exceptions.py b/scrapy/exceptions.py index 5330eab48..54a2cdbd6 100644 --- a/scrapy/exceptions.py +++ b/scrapy/exceptions.py @@ -1,10 +1,3 @@ -""" -Scrapy core exceptions - -These exceptions are documented in docs/topics/exceptions.rst. Please don't add -new exceptions here without documenting them there. -""" - from __future__ import annotations from typing import TYPE_CHECKING, Any diff --git a/scrapy/extensions/statsmailer.py b/scrapy/extensions/statsmailer.py index f05595806..25cc92dc6 100644 --- a/scrapy/extensions/statsmailer.py +++ b/scrapy/extensions/statsmailer.py @@ -1,9 +1,3 @@ -""" -StatsMailer extension sends an email when a spider finishes scraping. - -Use STATSMAILER_RCPTS setting to enable and give the recipient mail address -""" - from __future__ import annotations import warnings diff --git a/scrapy/settings/default_settings.py b/scrapy/settings/default_settings.py index 049921ba7..76cc85f19 100644 --- a/scrapy/settings/default_settings.py +++ b/scrapy/settings/default_settings.py @@ -1,18 +1,3 @@ -"""This module contains the default values for all settings used by Scrapy. - -For more information about these settings you can read the settings -documentation in docs/topics/settings.rst - -Scrapy developers, if you add a setting here remember to: - -* add it in alphabetical order, with the exception that enabling flags and - other high-level settings for a group should come first in their group - and pairs like host/port and user/password should be in the usual order -* group similar settings without leaving blank lines -* add its documentation to the available settings documentation - (docs/topics/settings.rst) -""" - import sys from importlib import import_module from pathlib import Path diff --git a/scrapy/signals.py b/scrapy/signals.py index 972f4fd60..383ef0245 100644 --- a/scrapy/signals.py +++ b/scrapy/signals.py @@ -1,10 +1,3 @@ -""" -Scrapy signals - -These signals are documented in docs/topics/signals.rst. Please don't add new -signals here without documenting them there. -""" - engine_started = object() engine_stopped = object() scheduler_empty = object() diff --git a/scrapy/utils/sitemap.py b/scrapy/utils/sitemap.py index 03b7bf3b1..2edf2d5ef 100644 --- a/scrapy/utils/sitemap.py +++ b/scrapy/utils/sitemap.py @@ -1,10 +1,3 @@ -""" -Module for processing Sitemaps. - -Note: The main purpose of this module is to provide support for the -SitemapSpider, its API is subject to change without notice. -""" - from __future__ import annotations import warnings diff --git a/scrapy/utils/trackref.py b/scrapy/utils/trackref.py index 22f9eadd0..7822f99b7 100644 --- a/scrapy/utils/trackref.py +++ b/scrapy/utils/trackref.py @@ -1,17 +1,3 @@ -"""This module provides some functions and classes to record and report -references to live object instances. - -If you want live objects for a particular class to be tracked, you only have to -subclass from object_ref (instead of object). - -This library has a minimal performance impact. - -.. note:: PyPy uses a tracing garbage collector, so objects may - remain in the ``live_refs`` longer than expected, even after they - go out of scope. If deterministic behavior is required, you may need - to explicitly trigger garbage collection or call ``trackref.live_refs.clear()``. -""" - from __future__ import annotations from collections import defaultdict