Move module docstring content to docs and remove remaining docstrings.

Migrate contributor notes, FTP handler details, trackref PyPy behavior,
SitemapSpider API stability warning, and STATSMAILER_RCPTS documentation
into the official docs before removing the last module-level docstrings.

Fixes #7699

Co-authored-by: Cursor <cursoragent@cursor.com>

docs/contributing.rst

@@ -308,6 +308,17 @@ In any case, if something is covered in a docstring, use the
 documentation instead of duplicating the docstring in files within the
 ``docs/`` directory.
 
+When adding a new setting to ``scrapy/settings/default_settings.py``:
+
+* add it in alphabetical order, with the exception that enabling flags and
+  other high-level settings for a group should come first in their group
+  and pairs like host/port and user/password should be in the usual order
+* group similar settings without leaving blank lines
+* add its documentation to :doc:`topics/settings`
+
+When adding a new exception or signal, document it in :doc:`topics/exceptions`
+or :doc:`topics/signals`, respectively.
+
 Documentation updates that cover new or modified features must use Sphinx’s
 :rst:dir:`versionadded` and :rst:dir:`versionchanged` directives. Use
 ``VERSION`` as version, we will replace it with the actual version right before

docs/topics/download-handlers.rst

@@ -378,6 +378,25 @@ This handler supports ``ftp://host/path`` FTP URIs.
 
 It's implemented using :mod:`twisted.protocols.ftp`.
 
+FTP connection parameters can be passed using :attr:`Request.meta
+<scrapy.Request.meta>`:
+
+* ``ftp_user`` (falls back to :setting:`FTP_USER`)
+* ``ftp_password`` (falls back to :setting:`FTP_PASSWORD`)
+* ``ftp_passive`` (falls back to :setting:`FTP_PASSIVE_MODE`)
+* ``ftp_local_filename``: if not given, file data is returned in
+  ``response.body``. If given, file data is saved to that local path (useful
+  for large files) and the local filename is also returned in ``response.body``.
+
+The built response uses HTTP-like status codes by default: ``200`` on success,
+``404`` when the file is not found (FTP code 550), or the corresponding FTP
+exception otherwise. The mapping from FTP return codes to response status codes
+is defined in the :attr:`~scrapy.core.downloader.handlers.ftp.FTPDownloadHandler.CODE_MAPPING`
+class attribute. The ``default`` key is used for unmapped codes.
+
+On successful requests (status 200), ``response.headers`` includes
+``Local Filename`` (when ``ftp_local_filename`` is used) and ``Size``.
+
 S3DownloadHandler
 -----------------
 

docs/topics/exceptions.rst

@@ -7,6 +7,11 @@ Exceptions
 .. module:: scrapy.exceptions
    :synopsis: Scrapy exceptions
 
+.. note::
+
+   When adding a new exception to ``scrapy.exceptions``, document it on this
+   page.
+
 .. _topics-exceptions-ref:
 
 Built-in Exceptions reference

docs/topics/leaks.rst

@@ -182,6 +182,13 @@ scrapy.utils.trackref module
 
 Here are the functions available in the :mod:`~scrapy.utils.trackref` module.
 
+.. note::
+
+   PyPy uses a tracing garbage collector, so objects may remain in
+   ``live_refs`` longer than expected, even after they go out of scope. If
+   deterministic behavior is required, you may need to explicitly trigger
+   garbage collection or call ``trackref.live_refs.clear()``.
+
 .. class:: object_ref
 
     Inherit from this class if you want to track live

docs/topics/request-response.rst

@@ -720,6 +720,8 @@ Those are:
 * :reqmeta:`download_warnsize`
 * :reqmeta:`download_timeout`
 * ``ftp_password`` (See :setting:`FTP_PASSWORD` for more info)
+* ``ftp_passive`` (See :setting:`FTP_PASSIVE_MODE` for more info)
+* ``ftp_local_filename`` (See :ref:`topics-download-handlers` for more info)
 * ``ftp_user`` (See :setting:`FTP_USER` for more info)
 * :reqmeta:`give_up_log_level`
 * :reqmeta:`handle_httpstatus_all`

docs/topics/settings.rst

@@ -2056,6 +2056,18 @@ Example:
 
 .. note:: This is a :ref:`pre-crawler setting <pre-crawler-settings>`.
 
+.. setting:: STATSMAILER_RCPTS
+
+STATSMAILER_RCPTS
+-----------------
+
+Default: ``[]``
+
+A list of email addresses to receive a message with scraping stats when a
+spider finishes. Used by the deprecated
+:class:`~scrapy.extensions.statsmailer.StatsMailer` extension, which is enabled
+when this setting is non-empty.
+
 .. setting:: STATS_CLASS
 
 STATS_CLASS

docs/topics/signals.rst

@@ -16,6 +16,10 @@ deliver the arguments that the handler receives.
 You can connect to signals (or send your own) through the
 :ref:`topics-api-signals`.
 
+.. note::
+
+   When adding a new signal to ``scrapy.signals``, document it on this page.
+
 Here is a simple example showing how you can catch signals and perform some action:
 
 .. code-block:: python

docs/topics/spiders.rst

@@ -787,6 +787,11 @@ SitemapSpider
     SitemapSpider allows you to crawl a site by discovering the URLs using
     `Sitemaps`_.
 
+    .. note::
+
+       The :mod:`scrapy.utils.sitemap` module provides sitemap parsing support
+       for this spider. Its API is subject to change without notice.
+
     It supports nested sitemaps and discovering sitemap urls from
     `robots.txt`_.
 

scrapy/core/downloader/handlers/_httpx.py

@@ -1,5 +1,3 @@
-"""``httpx``-based HTTP(S) download handler. Currently not recommended for production use."""
-
 from __future__ import annotations
 
 import ipaddress

scrapy/core/downloader/handlers/ftp.py

@@ -1,33 +1,3 @@
-"""
-An asynchronous FTP file download handler for scrapy which somehow emulates an http response.
-
-FTP connection parameters are passed using the request meta field:
-- ftp_user (optional, falls back to FTP_USER)
-- ftp_password (optional, falls back to FTP_PASSWORD)
-- ftp_passive (optional, falls back to FTP_PASSIVE_MODE) sets FTP connection passive mode
-- ftp_local_filename
-        - If not given, file data will come in the response.body, as a normal scrapy Response,
-        which will imply that the entire file will be on memory.
-        - if given, file data will be saved in a local file with the given name
-        This helps when downloading very big files to avoid memory issues. In addition, for
-        convenience the local file name will also be given in the response body.
-
-The status of the built html response will be, by default
-- 200 in case of success
-- 404 in case specified file was not found in the server (ftp code 550)
-
-or raise corresponding ftp exception otherwise
-
-The matching from server ftp command return codes to html response codes is defined in the
-CODE_MAPPING attribute of the handler class. The key 'default' is used for any code
-that is not explicitly present among the map keys. You may need to overwrite this
-mapping if want a different behaviour than default.
-
-In case of status 200 request, response.headers will come with two keys:
-    'Local Filename' - with the value of the local filename if given
-    'Size' - with size of the downloaded data
-"""
-
 from __future__ import annotations
 
 import re

scrapy/exceptions.py

@@ -1,10 +1,3 @@
-"""
-Scrapy core exceptions
-
-These exceptions are documented in docs/topics/exceptions.rst. Please don't add
-new exceptions here without documenting them there.
-"""
-
 from __future__ import annotations
 
 from typing import TYPE_CHECKING, Any

scrapy/extensions/statsmailer.py

@@ -1,9 +1,3 @@
-"""
-StatsMailer extension sends an email when a spider finishes scraping.
-
-Use STATSMAILER_RCPTS setting to enable and give the recipient mail address
-"""
-
 from __future__ import annotations
 
 import warnings

scrapy/settings/default_settings.py

@@ -1,18 +1,3 @@
-"""This module contains the default values for all settings used by Scrapy.
-
-For more information about these settings you can read the settings
-documentation in docs/topics/settings.rst
-
-Scrapy developers, if you add a setting here remember to:
-
-* add it in alphabetical order, with the exception that enabling flags and
-  other high-level settings for a group should come first in their group
-  and pairs like host/port and user/password should be in the usual order
-* group similar settings without leaving blank lines
-* add its documentation to the available settings documentation
-  (docs/topics/settings.rst)
-"""
-
 import sys
 from importlib import import_module
 from pathlib import Path

scrapy/signals.py

@@ -1,10 +1,3 @@
-"""
-Scrapy signals
-
-These signals are documented in docs/topics/signals.rst. Please don't add new
-signals here without documenting them there.
-"""
-
 engine_started = object()
 engine_stopped = object()
 scheduler_empty = object()

scrapy/utils/sitemap.py

@@ -1,10 +1,3 @@
-"""
-Module for processing Sitemaps.
-
-Note: The main purpose of this module is to provide support for the
-SitemapSpider, its API is subject to change without notice.
-"""
-
 from __future__ import annotations
 
 import warnings

scrapy/utils/trackref.py

@@ -1,17 +1,3 @@
-"""This module provides some functions and classes to record and report
-references to live object instances.
-
-If you want live objects for a particular class to be tracked, you only have to
-subclass from object_ref (instead of object).
-
-This library has a minimal performance impact.
-
-.. note:: PyPy uses a tracing garbage collector, so objects may
-    remain in the ``live_refs`` longer than expected, even after they
-    go out of scope. If deterministic behavior is required, you may need
-    to explicitly trigger garbage collection or call ``trackref.live_refs.clear()``.
-"""
-
 from __future__ import annotations
 
 from collections import defaultdict