feat: Adding documentation to feature

docs/topics/downloader-middleware.rst

@@ -856,6 +856,21 @@ OffsiteMiddleware
    :attr:`~scrapy.Spider.allowed_domains` attribute, or the
    attribute is empty, the offsite middleware will allow all requests.
 
+   If the spider defines a :attr:`~scrapy.Spider.disallowed_domains`
+   attribute, any request whose host name matches one of the domains in
+   that list (or their subdomains) will be filtered out, regardless of
+   :attr:`~scrapy.Spider.allowed_domains`. This is useful when you want
+   to allow most domains but block a few specific ones.
+
+   If a request matches both :attr:`~scrapy.Spider.allowed_domains` and
+   :attr:`~scrapy.Spider.disallowed_domains`, it will be filtered out
+   (i.e. :attr:`~scrapy.Spider.disallowed_domains` takes precedence).
+
+   Both :attr:`~scrapy.Spider.allowed_domains` and
+   :attr:`~scrapy.Spider.disallowed_domains` must contain valid domain
+   names only (not URLs or domains with ports). Invalid entries will cause
+   the spider to close with reason ``invalid_domain_configuration``.
+
    .. reqmeta:: allow_offsite
 
    If the request has the :attr:`~scrapy.Request.dont_filter` attribute set to

docs/topics/spiders.rst

@@ -74,6 +74,18 @@ scrapy.Spider
        Let's say your target url is ``https://www.example.com/1.html``,
        then add ``'example.com'`` to the list.
 
+   .. attribute:: disallowed_domains
+
+       An optional list of strings containing domains that this spider is
+       not allowed to crawl. Requests for URLs belonging to the domain names
+       specified in this list (or their subdomains) will be filtered out when
+       :class:`~scrapy.downloadermiddlewares.offsite.OffsiteMiddleware` is
+       enabled.
+
+       This is useful when you want to allow all domains except a few
+       specific ones. For example, to block requests to ``ads.example.com``,
+       add ``'ads.example.com'`` to the list.
+
    .. autoattribute:: start_urls
 
    .. attribute:: custom_settings

tests/test_downloadermiddleware_offsite.py

@@ -135,9 +135,7 @@ def test_invalid_domains_closes_spider(caplog):
     mock_engine = AsyncMock()
     crawler.engine = mock_engine
     with (
-        patch(
-            "scrapy.downloadermiddlewares.offsite._schedule_coro"
-        ) as mock_schedule,
+        patch("scrapy.downloadermiddlewares.offsite._schedule_coro") as mock_schedule,
         caplog.at_level(logging.ERROR),
     ):
         mw.spider_opened(crawler.spider)