diff --git a/docs/_ext/scrapydocs.py b/docs/_ext/scrapydocs.py index 8358a01af..a7e86f6a5 100644 --- a/docs/_ext/scrapydocs.py +++ b/docs/_ext/scrapydocs.py @@ -4,18 +4,8 @@ def setup(app): rolename = "setting", indextemplate = "pair: %s; setting", ) - app.add_crossref_type( - directivename = "topic", - rolename = "topic", - indextemplate = "pair: %s; topic", - ) app.add_crossref_type( directivename = "signal", rolename = "signal", indextemplate = "pair: %s; signal", ) - app.add_crossref_type( - directivename = "exception", - rolename = "exception", - indextemplate = "pair: %s; exception", - ) diff --git a/docs/experimental/images.rst b/docs/experimental/images.rst index 91dd187a9..1133046d9 100644 --- a/docs/experimental/images.rst +++ b/docs/experimental/images.rst @@ -327,7 +327,7 @@ MediaPipeline The return value of this method is used as output of pipeline stage. :meth:`item_completed` can return item itself or raise - :exception:`DropItem` exception. + :exc:`~scrapy.core.exceptions.DropItem` exception. Default returns item diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst index e8cd8158b..ca9e329cb 100644 --- a/docs/topics/downloader-middleware.rst +++ b/docs/topics/downloader-middleware.rst @@ -83,8 +83,8 @@ will then be passed to the original callback. If the new request doesn't have a callback, the response downloaded will be just passed to the original request callback. -If returns an :exception:`IgnoreRequest` exception, the entire request will be -dropped completely and its callback never called. +If returns an :exc:`~scrapy.core.exceptions.IgnoreRequest` exception, the +entire request will be dropped completely and its callback never called. .. method:: process_response(request, response, spider) @@ -94,14 +94,14 @@ dropped completely and its callback never called. ``spider`` is a BaseSpider object ``process_response()`` should return a Response object or raise a -:exception:`IgnoreRequest` exception. +:exc:`~scrapy.core.exceptions.IgnoreRequest` exception. If returns a Response (it could be the same given response, or a brand-new one) that response will continue to be processed with the ``process_response()`` of the next middleware in the pipeline. -If returns an :exception:`IgnoreRequest` exception, the response will be -dropped completely and its callback never called. +If returns an :exc:`~scrapy.core.exceptions.IgnoreRequest` exception, the +response will be dropped completely and its callback never called. .. method:: process_download_exception(request, exception, spider) @@ -196,6 +196,8 @@ HttpCacheMiddleware * :setting:`HTTPCACHE_EXPIRATION_SECS` - how many secs until the cache is considered out of date +.. _topics-dlmw-robots: + RobotsTxtMiddleware ------------------- diff --git a/docs/topics/exceptions.rst b/docs/topics/exceptions.rst index 0fa6b524a..f6bc0406f 100644 --- a/docs/topics/exceptions.rst +++ b/docs/topics/exceptions.rst @@ -14,44 +14,44 @@ Built-in Exceptions reference Here's a list of all exceptions included in Scrapy and their usage. -.. exception:: DontCloseDomain - DontCloseDomain --------------- +.. exception:: DontCloseDomain + This exception can be raised by any handler of the :signal:`domain_idle` signal to avoid the domain from being closed at this time, and wait for the next idle state. -.. exception:: DropItem - DropItem -------- +.. exception:: DropItem + The exception that must be raised by item pipeline stages to stop processing an Item. For more information see :ref:`topics-item-pipeline`. -.. exception:: HttpException - HttpException ------------- +.. exception:: HttpException + This exception is raised by the downloader when a non-200 response has been downloaded. -.. exception:: IgnoreRequest - IgnoreRequest ------------- +.. exception:: IgnoreRequest + This exception can be raised by the Scheduler or any downlaoder middleware to indicate that the request should be ignored. -.. exception:: NotConfigured - NotConfigured ------------- +.. exception:: NotConfigured + This exception can be raised by some components to indicate that they will remain disabled. Those component include: @@ -62,10 +62,10 @@ remain disabled. Those component include: The exception must be raised in the component constructor. -.. exception:: NotSupported - NotSupported ------------ +.. exception:: NotSupported + This exception is raised to indicate an unsupported feature. diff --git a/docs/topics/extensions.rst b/docs/topics/extensions.rst index 8a8ac01b6..af1c2ecd3 100644 --- a/docs/topics/extensions.rst +++ b/docs/topics/extensions.rst @@ -89,9 +89,9 @@ Writing your own extension is easy. Each extension is a single Python class which doesn't need to implement any particular method. All extension initialization code must be performed in the class constructor -(``__init__`` method). If that method raises the :exception:`NotConfigured` -exception, the extension will be disabled. Otherwise, the extension will be -enabled. +(``__init__`` method). If that method raises the +:exc:`~scrapy.core.exceptions.NotConfigured` exception, the extension will be +disabled. Otherwise, the extension will be enabled. Let's take a look at the following example extension which just logs a message everytime a domain/spider is opened and closed:: diff --git a/docs/topics/item-pipeline.rst b/docs/topics/item-pipeline.rst index abd48066e..0a57bff8d 100644 --- a/docs/topics/item-pipeline.rst +++ b/docs/topics/item-pipeline.rst @@ -29,8 +29,8 @@ single Python class that must define the following method: This method is called for every item pipeline component and must either return a :class:`~scrapy.item.Item` (or any descendant class) object or raise a -:exception:`DropItem` exception. Dropped items are no longer processed by -further pipeline components. +:exc:`~scrapy.core.exceptions.DropItem` exception. Dropped items are no longer +processed by further pipeline components. Item pipeline example diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst index aaf34d942..bd093ee17 100644 --- a/docs/topics/settings.rst +++ b/docs/topics/settings.rst @@ -769,7 +769,7 @@ Default: ``False`` Scope: ``scrapy.contrib.downloadermiddleware.robotstxt`` If enabled, Scrapy will respect robots.txt policies. For more information see -:topic:`robotstxt` +:ref:`topics-dlmw-robots` .. setting:: SCHEDULER diff --git a/docs/topics/signals.rst b/docs/topics/signals.rst index 7866df72b..e08750763 100644 --- a/docs/topics/signals.rst +++ b/docs/topics/signals.rst @@ -79,9 +79,9 @@ order. :type spider: :class:`~scrapy.spider.BaseSpider` object If any handler of this signal handlers raises a - :exception:`DontCloseDomain` the domain won't be closed this time and will + :exc:`DontCloseDomain` the domain won't be closed this time and will wait until another idle signal is sent. Otherwise (if no handler raises - :exception:`DontCloseDomain`) the domain will be closed immediately after + :exc:`DontCloseDomain`) the domain will be closed immediately after all handlers of ``domain_idle`` have finished, and a :signal:`domain_closed` will thus be sent. @@ -106,7 +106,7 @@ order. :param item: is the item scraped :type item: :class:`~scrapy.item.Item` object - :param spider: the spider which scraped the item + :param spider: the spider which scraped the item :type spider: :class:`~scrapy.spider.BaseSpider` object :param response: the response from which the item was scraped @@ -121,7 +121,7 @@ order. :param item: the item which passed the pipeline :type item: :class:`~scrapy.item.Item` object - :param spider: the spider which scraped the item + :param spider: the spider which scraped the item :type spider: :class:`~scrapy.spider.BaseSpider` object :param output: the output of the item pipeline. This is typically the @@ -132,17 +132,18 @@ order. .. function:: item_dropped(item, spider, exception) Sent after an item has been dropped from the :ref:`topics-item-pipeline` - when some stage raised a :exception:`DropItem` exception. + when some stage raised a :exc:`~scrapy.core.exceptions.DropItem` exception. :param item: the item dropped from the :ref:`topics-item-pipeline` :type item: :class:`~scrapy.item.Item` object - :param spider: the spider which scraped the item + :param spider: the spider which scraped the item :type spider: :class:`~scrapy.spider.BaseSpider` object - :param exception: the exception (which must be a :exception:`DropItem` - subclass) which caused the item to be dropped - :type exception: :exception:`DropItem` exception + :param exception: the exception (which must be a + :exc:`~scrapy.core.exceptions.DropItem` subclass) which caused the item + to be dropped + :type exception: :exc:`~scrapy.core.exceptions.DropItem` exception .. signal:: request_received .. function:: request_received(request, spider, response)